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FOREWORD 



When the U.S. Office of Education was 
chartered in 1867, one charge to its 
commissioners was to determine the nation's 
progress in education. The National 
Assessment of Educational Progress (NAEP) 
was initiated a century later to address, in a 
systematic way, that charge. 

Each year since 1969, National Assessment 
has gathered information about levels of edu- 
cation achievement across the country and 
reported its fmdinjp to the nation. NAEP sur- 
veys the education attainments of 9-year-o!ds, 
iS year-olds and 17-year-oIds in 10 learning 
areas: art, career and occupational develop- 
ment, citizenship, literature, mathematics, 
music, reading, science, social studio and 
writing. Different learning areas are assessed 
every year, and all areas are periodically re- 
assessed in order to measure possible changes 
in education ^ achievement. National Assess- 
ment has interviewed and tested more tlian 
810,000 young Americans since 1969. 

Leaming-area assessments evolve Trom a 
consensus process. Each assessment is the 
product of several years of work by a great 
many educators, scholars and lay persons 
from all over the nation. Initially, these 
people design objectives for each subject area, 
proposing general goals they feel Americans 



should be achieving in the course of their edu- 
cation. After careful reviews, these objectives 
are given to exercise (item) writers, whose 
task it is to create measurement instruments 
appropriate to the objectives. 

When the exercises have passed extensive 
reviews by subject-matter specialists, measure- 
ment experts and lay persons, they are admin- 
istered to probability samples. The people 
who compete these samples are chosen in 
such a way that the r^ults of their assessment 
can be generalized to an entire national popu- 
lation. That is, on the basis of the perform- 
ance of about 2,500 9-year-olds on a given 
exercise, we can make generalizations about 
the probable performance of all 9-year-olds in 
the nation. 

After assessment data have been collected, 
scored and analyzed, National Assessment 
publishes repcrts to disseminate the results as 
widely as pebble. Not all exercises are 
released for publication. Because NAEP will 
readminister some of the same exercises in the 
future to determine whether the performance 
level of Americans has increased, remained 
stable or decreased, it is e^ntial that they 
not be released in orde^ to presme the integ- 
rity of the study. 
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IMTROOUCTION 



The National Assessment of Educational 
Projjress (NAEP) has completed two 
assessments of mathematics, the first 
conducted in 1 972^-73 and the second during 
1977—78. Each as&Msment surveyed the 
mathematics achievement of American 9*, 13- 
and 17-year-olds» using a deeply stratified, 
multistage probability sample design. This 
report documents procedures used in the 
1977-^78 mathematics assessment and also 
describes changes in procedures between the 
assessments. 

To measure changes in performance 
between 1972-73 and 1977-78. 
approximately half of the exercises assessed in 
the first assessment were reassessed in the 
second under almost identical administrative 
conditions. To mea£ure the status of 
mathematics ach '^ment in 1977—78. Na- 
tional Assessment .isultants revised the ob- 
jectives used in lie first assessment and 
developed additional exercises to provide 
complete coverage of the revised objectives. 

Approximately 17,000 9'year-olds, 27,000 
13-year-olds and 27,000 17-year-olds partici- 
pated in the 1977-78 mathematics assess- 
ment. National Assessment reports results for 
^t-oups of students, not individuals. Therefore, 
it is not necessary for each student to respond 
to every item (exercise).' Each respondent 
completed only one item booklet of about 45 
minutes in length. Between 2,100 and £,500 
student* responded to each booklet. In 
1977—78, there were 7 exercise booklets for 
9-ye!ur-oIds, 11 booklets for 13-year-olds and 



Natiatui AiammaRt uaM the term "•xcrcUe" to 
mMn an UMwmcnt item. The temu "exerciee" ind 
"item" are used iaterchangegbly in this report. 



12 booklets for 17-year-olds. 

In each assessment, 13-year-olds were 
assessed in October through December; 9- 
year-olds in January and February; ana 17- 
year-olds in March and April. Thus, the 
amouiit of school experience for each age 
group was approximatei. the same in each 
ass^ment. 

The exercises for each assessment were 
administered by a professional data collection 
staff to minimize the burden on participating 
schools and to maximize uniformity of assess- 
ment conditions. Instructions arid items were 
recorded on a paced audio tape and played 
back to students to reduce the potential 
effect of reading difficulties ahd to insure that 
all students moved throu^ the packages at 
the same speed. 

Each item included one or more item parts; 
these item parts were analyzed separately. 
Over 900 item parts were mcluded in the 
1977—78 mathematics assessment. About half 
the items wc.*e multiple-choice and about half 
were open^ndad, or fr6e response, items. 
Changes in achievement between 1972—73 
and 1977—78 were based upon 55 item parts 
for 9-yearKJlds, 77 item parts for 13-year-olds 
and 102 item parts for 17-year-oIds. Approx- 
imately 20% of the items used to measure 
change were multiple-choice; the remaining 
80% were open-ended. 

Multiple-choice items were scored by an 
optical scanning machine; open-ended items 
were hand-scored by trained scorers using 
scoring guides, which defined aitegories of 
acceptable and unacceptable responses. These 
scoring guides were developed following field 
testing of the items and then revised and 
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refined during nn^eipt of initial asse^^sment 
data. 

National Assensment reports estimate per^ 
centa^es of correct responses for single items. 
When a report indicatt»s that of the 

17-year-oids ^ave a correct response/' it 
means that an estimated 85% of the 17-year- 
ohis would have given a correct response if all 
the 17 year*oIds in schools across the country 
had been assessed. In addition to reporting 
national results, National Assessment provides 
data on the f performance of various popula- 
tion suhfjroups within the national popula- 
tion, defined by sex, race, region of the 
country, size and type of community lived in, 
and level of parental education. For the 
1977" 78 mathematics assessment, some data 
are available on the performance of 17-year- 
old students with varying levels of math- 
ematics course- taking experience,* National 



' Svv ApjH^ndix B for definitions of the subgroups 
rt»pi)fted upon. 



Assessment aggregates percentages of success 
on various sets of items to provide data on 
changes in performance between assessments 
and on the differential performance of 
population subgroups. 

The following chapters describe specific 
procedures used to develop objectives and 
exerciiies, draw the ass^ment sample, pre^ 
pare materials for the assessmrnl, administer 
and score the items and analyse results. This 
information ^ould prove useful to those 
wishing to replicate National Assessment pro- 
cedures and to r^archers and data usei^ with 
an interest in how the data were obtained. 
Materials providing information about the 
results of the assessment are described in 
Chapter 8. 
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CHAPTER 1 



OBJECTIVES 



The primary goal of the National Assess- 
ment of Educational Progress (N^EP) is to 
report, on thj- current education status of 
young Americans and to monitor any changes 
in achievement over time. For each learning 
area to be assessed, NAEP asks consultants to 
develop objectives that define the subject' 
area; specify knowledge, skills and attitudes 
to be assessed; and serve as guides for exercise 
developers. 

Education in America is a coUaborative 
enterprise involving a great many people with 
widely differing philosophies. Providing in- 
formation about education nationwide would 
be considerably easier if there were consensus 
bout the moans and ends of American 
^ucation, but the fact is that Americans have 
conflicting and sometimes contradictory 
values reganiing the goals of education and 
the means for achieving them. To develop an 
assessment that is truly national in scope and 
takes into account the diversity of curricula, 
values and goals across the country, STxtiontl 
Assessment employs a consensus process for 
developing objectives, involving represen- 
tatipn of many different groups of people. 

Several types of consultants helped develop 
the 1977-78 mathematics objectives. College 
and university specialists in mathematics in- 
sured that the objectives included important 
concepts that the schools should be teaching. 
Educators, including classroom teachers, cur- . 

^iiculum supervisors and per«}ns involved in 
teacher education, made sure that the ob- 
jectives included concepts, skills and attitudes 
that the schools should be teaching and thpse 
that they presently are teaching. Concerned 

' citizens, parents and other interested lay 
parsons had to agree that the objectives wgre 



REDEVELOPMENT 



important for young people to achieve, were 
free of education jargon and were not biased 
or offensive to any groups. Consultants were 
representative of the different regions of the 
country, minority groups, both sexes, and 
various types of communities, age levels and 
education philosophies. 

The objectives for the 1972-73 math- 
ematics assessment were' developed by an 
outside contractor, utilizing the types of 
consultants described above. Objectives for 
the second assessment were developojj-^ 
throu^ conferences organized and conducted 
by the National Assessment staff. This pro- 
cedure increased interaction between assess- 
ment staff and consultants, involved more 
professionals in. the areas of math<'matics and 
education, allowed greater flexibility and was 
more efficient in terms of cost and time.* 

Redevelopment of the mathematics ob- 
jectives for the second assessment was built 
upon the framework established by the ob- 
jectives for the first assessment., The 1972-73 
mathematics objectives were originally or- . 
ganized ipto a three-dimensional scheme: (1) 
uses of mathematics, (2) mathematical con- 
tent and (3) cognitive ability levels. The first 
dimension — defined as social, technical or 
academic uses — proved difficult to apply in 
guiding exercise development and so was 
abandoned as a functioning dimension of the 
objectives. The remaining two dimensions 
included 17 content categories and 6 cog- 



* For a more complete deicription of tfad procedur*. 
ttted In mtevelopijig the objectiTet, eee Matktmatics 
ObjecthteM, Second Attinment (197S). 
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. nitivt» abiiity calejjories. 

977 - 78 mathematics objectives were 
also (,*r^anized in a content-by-proeess (or 
•oKnitive ability) matrix. This matrix re- 
sembled the classification scheme developed 
fur the first a&s^^sment, but fewer and more 
inc'iusive content and process categories were 
includfe*cJ in the matrix for the second assess- 
ment. Ihe second assessment objectives in- 
chidtKi five content, categories and four 
cognitive-process ieveis* (Exhibit 1). 

In de'-eloping the mathematics assessment, 
the consultants felt that several topics not 
included in the matrix were still highly 
important, to assess. Sorne of these topics 
looked toward future emphases in math- 
ematics eiiucation: use of the hand-held cal- 
culator, computer literacy and 



^ For a description of the cnntVnt and process levels, 
see Mathematics Objectives, Second Asse&i»ment 



computer-related mathematics. Exercises ad- 
dressing these topics will provide baseline data 
against which to measure the future impact of 
these topics in the schools. The me lurement 
of attitudes toward mathematics also received 
considerable attention. Attitudes were not 
measured during the first assessment because \ 
of the difficulty of developing adequate meas- 
ures. However, consultants for the second 
assessment strongly encouraged the develop- 
ment of attitudina! exercises, so a con- 
centrated effort was made to include such 
measures. 

After consultants develop^ the initial ob- 
jectives. National Assessment formed an ad- 
visory board to give direction and advice to 
NAEP staff for further refinement of the 
objectives and development of the assessment. 
The six -member advisory board included 
three mathematics educators, two math- 
ematicians and a mathem;atics teacher (see 
Appendix A for members). This board was. 
instrumental in organizing the final set of 
objectives, planning the development of exer- 
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EXHIBIT t. Contani by Process Matrix for ths 1977-7S Asstfsmsnt of Mathematics 

CONTENT 





Numbers 


VariabTtt 




Maasufe^ 


Otfwr 




and 


and 


IShapt, 




Topics 




Nurtitri- 


Rtfation* 


Sizf and 






tioft 


sMpt 


Poaition) 




Mathematicai 






• 1 






knowledge 












Mathematical 








0 




skrli 












Mathf»matical 












understanding 












Mathemattcai 












application 











1-1 



ERIC 



cises, selecting the fmai exerciws and planning 
the subsequent report topics. . 

The objectives were also reviewed by a 
panel of lay persons to insure that they 
covered topics generally considered to be 
important and that they were not biased. In 
addition, they were looked upon as guides to 
structure reporting. National Assessment staff 
and the advisory board felt that reporting of 
results would be better organized and more 
comprehensive if a reporting scheme existed 
prior to exercii;e development. Accordingly, 
the advisory board recommended structuring 
the reporting by cognitive-proc^ levels, and 
they devised a set of qurations that relate to 
each of the four cc^itive processes (Exhibit 
2). 



EXHIBIT 2, Qutfstiom Used To Structurt 
Rtpofting of th« 1977-78 Matiiematics Assessment 

i . Matbematica! knowfedge 

A, How well can students recall and re- 
Qpgnize facts, definitions and symbols? 

i>. Mathenaifca! skit! 

A. How well can students perform com- 
putations, including computations with 
whole numbers, integers, fractions, de- 
cimais, percents, ratios and proportions? 

B. How weU can students make measure- 
ments? 

C. How well can students read graphs and 
tables? 



D. How well can students-perform ^metric 
manipulations fike constructions and 
spatia! visualizations? 

E. How well can students perform algebraic 
manipulations? 

F. How well can students estimate the 
answers to computations and measure- 
ments? 

III. Mathematical understanding 

A. How well can students translate a verba! 
statement into symbols or a figure, and 
vice versa? 

B. How well do students understand math- 
ematical concepts anc* principles? 

IV. ; Mathematical application 

A. How well can students solve typical text- 
book problems? 

B. How well can students solve r^snroutine 
problems? 

C. How well can students estimate the 
answers to problems? 

O. How well can students use mathematics 
in reasoning and making jiKigmer^ts? 

. While National Assessment cannot supply 
definitive answers to such questions (deter- 
mining how "well" students perform is a 
subjective judgment), it can provide data to 
assist individual readers in making their bwn 
judgments about tbe^ questions. 
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CHAPTER 2 



DEVELOPMENT OF EXERCISES 



Exercises were developed to provide in- 
formation about achievement ieveis for the 
various cells of the objective matrix and to 
provide data that could be used in answering 
the questions shown in Exhibit 2. Each 
exercise wps designed so that its r^ults could 
either be used alone, .as an indiotor of 
performance on a specific task, or used in 
conjunction with results from other exercise 
to give a more general picture of achievement 
levels. 



Exercise Cteveiopmmit 

In developing the objectives, groups of 
consultants and the advisory board weighted 
the relative importance of the various ceils in 
the objectives matrix. Theie weights were 
used -to determine relative amounts of time to 
be devoted to assessing each cell and were 
utilized in the final selection of exercises. 

The 1972—73 exercises that we^ re- 
assessed in ;^97,7-78 to measure change were 
reclassified in the ceils of the 1977—78 
objectives matrix. This allowed the same 
reporting scheme — cognitive-process levels — 
to be used for both change and 1977—78 
status reporting. In addition, this meant that 
1977—78 results for the items used to meas- 
lire change could also be used as a' part of the 
1977-78 reporting. 

Most item writing was done by groups of 
people active in the fields of mathematics and 
mathematics education. Approximately 150 
people served as item developers. Rather than 
addressing specific ceils of the objectives 
matrix, the groups worked primarily with the 



process dimension of the matrix, with each 
group developing a set of items that addressed 
one or two of the questions from the list of 
cognitive-process questions shown in Exhibit 
2. Individuals generated items, which were 
then reviewed and revised by the members of 
the group. Reviewers ponsidered age-level 
appropriateness, accuracy of content, how 
well the item measured a question or ob- 
jective, and readability. Exercises passing the 
group review were edited by the National 
Assessment staff to fit NAEP format and 
technical requirements, 



Fie^ld Tryouts 

The exercises produced by the writing 
groups for the 1977-78 mathematics assess- 
ment were field tested in schools across the 
country to discover potential problems in 
wording, directions or administration pro- 
cedures and to coUect. item statistics, tiining 
information and scoring information. 
"Tryout" schools were selected to represent 
high- and low-income communities as well as 
more typical communities. The tryouts were 
administer^ to students in at least four 
classrooms (approximately 100 students) at 
each of the ages assessed. So ^t the tryouts 
would closely simulate actual assessment field 
procedures, the students recorded their an- 
swers in the test booklets; directions ands 
questions weresesd to students from an audio 
tape; and National Asseaanent staff members, 
rather than classroom- teachers, administ^ed' 
the test. The completed tryout packages were 
then scored and submitted to item analysis. 
The item-analjrsis statistics, as well ' as the 
administrators' reports of any field problems. 



helfHKi both staff and eunsuitants to evaluate 
and revise the exercises. Revised exercises 
were generally field tested again. 

After the initial exercise pool was develop- 
ed by the writing groups working with the 
process dimension of the matrix, tliree writing 
teams reviewed the exercises and accompany- 
ing tryout data (organized by content cate- 
gories) to insure that content areas were 
adequately covered. These teams edited exist- 
ing exercises a?id generated new ones, which 
again, were field tested. 



Exercise Reviews and Final SeiecticMi 

Each Item selected to be considered for 
mclusion in the final 1977—78 assessment 
pool of items was reviewed in a series of 
conferences by at least 25 different con- 
sultants: mathematicians, mathematics edu- 
cators, classroom teachers and interested lay 
citizens. Exercises for each age group were 
reviewed for af^^ropriateness by teachers who 
taught students at that age. Lay citizens, 
representing a variety of occupations and 
•interests, also reviewed the exercises, checking 



for sex or racial bias and considering the 
general importance of each exercise. 

The final exercise selection for the 
1977—78 assessment was approved by the 
advisory board, which us€Ki assigned cell 
weights to decide how many exercises to 
include in the assessment of each objective. 
The nujnber of exercises that could be assess- 
ed was limited by the amount of testing time 
available at each age: 245 minutes at age 9 (7 
booklets), 385 minutes at age 13 (11 book- 
lets) and 420 minutes at age 17 (12 booklets). 
Since there was not enough testing time to 
include all of the exercises selected. National 
Ass^ment staff, with the concurrance of the 
advisory board, ^ deleted a' small number of 
exercises. ^ " \ 

An item could have one or more parts. For 
the mathematics as^^ments, each item part 
was analyzed as a separate unit of data. Table 
1 shows the approximate number of item 
parts included iti each content area and in 
each cognitive-process level by age. 

' In developing the mathematics ass^ment. 
National Assessment included items th^t 



TABLE 1. Number of Item Parts A$semdfor Ei^ Cogniti\)«-Procms 
Level and Content Category, by Age 

Age 9 Age 13 Age 17 
Cognltlve-Prooefi Level ^ 



Knowledge 


161 


147 


140 


Skill 


' 137 


272 


273 


Understanding 


44 


. 108 


10S 


Applications 


44 


106 


136 


* 


Cofitsnt Cattgory 






Numbers and numeration 




327 


300 


Variables and rsiationships 


24 ' 


60 


108 


Geometry 


44 


100 


106 


Measurement 


49 


64 


61 


Other topics 


43 


82 


79 


Attitudes 


118 


207 


203 


Required use of calculator 


37 


41 


48 


Workins with computers 




• 4 

> 


4 



*No item parts a^e^d at this age: 
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dejiart somewhat from the traditional math- 
ematii^ •'paper and j>encir' test item. A 
number of items at each age required the use 
of apparatus such as a compass, an English 
ruler or a metric ruler, which were supplied to 
the students. One booklet at each age re- 
quired the use of a simple^ four-function 
electronic hand calculator, yvhich was also 
supplied. Some of the exercise included in 
the calculator booklets also appeared in book< 
lets for which the calculator was not used, 
thus enabling comparison of performance 
with and without a calculator. 



The -second assessment of mathematics 
contained fewer ooenended items than the 
first assessment. While open-ended exercises 
can provide valuable diagnostic information 
on errors commonly made by the students, 
they are considerably more expensive and 
more complicated to score than 
multiple-choice items. In many cases, care- 
fully designed multiple-choice questions can 
elicit sufficient information. All open^nded 
exercise from the first ass^ment used to 
measure change were readministered in their 
original format. 



CHAPTER 3 



PREPARATION OF 



OvBfview of Padcajpng 

Following the selection" of mathematics 
exercises to be included in the assessment. 
National Assessment staff grouped and se^ 
quenced items into ex&cdse booklets Since 
students at different ages deceived somewhat 
diff«ent sets of exerdsfe, booklets were 
constructed s^arately for each age level. 
Thus, exerdsies for 9-year-olds were not se^ 
•quenced in the same order as those for 
13-year-olds, and so forth. 

The following constraints were observed in 
preparing exercise booklets: ' 

1. Each booklet contained exercises of 
varying difficulty so that students would 
not become bored by^ many easy exer- 
cises or discouraged by many difficult 
exercises. 

2. Each booklet' contained exercises meas- 
uring a variety of cognitive-process levels 
and coittent categoric. 

3. Exercises could not cue other exercises. 
In other words, the answer to one exer- ' 
else could not be contained in another 
exercise in the same booklet 

4. Each booklet was timed so that it would 
take no more than 45 mLiutes — the 
length of a typical class period — of a 
stiident*s time. Booklets contained 
approximately 30-35 ^minutes of exer- 
cise time and. an additional 10-15 
minutes of iatioductory. material, in- 
itnicticau and backgrcHind questions. 



MATERIALS 



5. Booklets were designed to be, insofar ^ 
possible, parallel with respect to the 
number of different objectives measured 
and difficulty levels. Items measuring a 
particular objective were scattered 
throu^iout the booklets so that many 
different students would respond to 
questions related to a particular ob^- 
jective. 

National Assessment makes every effort to 
minimize di^culti^ connected with the test- 
ing situation so that results vrili be, as nearly 
as possible, an accurate reflection jof what 
students know. and can do. For example, 
students marked their answers directly in the 
assessment booklets, not on separate answer 
sheets. It was felt that tliis proc^ure would 
reduce possibilities for errors in marking 
answer sheets, ^ especially for the younger 
students. To minimize guesang, studenfe were 
encouraged to write "I don't know" on the 
answer line for open-ended questions or to 
select the "I dpnt know" response option 
included with ^ch multiple-choice itent if 
they felt they did not know the answer to a 
qt^stion. . 

Paced audio tapes were prepared for each 
exercise booklet. Instructions, most of the 
written portions of joi exerdse stimuhis and 
response options were read aloud -to minimize 
the ef£ect of any reading difficulties and to 
insure, that all students moved tUrou^ the 
booklets at the same speed. In adiitiim, the 
use of Uim helped to insure unifc rm assess- 
ment comSitions acrou the country. IXiriag 
the fidd testing of the exsdsea, adminis- 
tk^tass had determined the tts» needed for 
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most of the students to respond to an item. 



Differences in Packaging: 
1972-73 and t977-78 

National Assessment makes every effort to 
make asst'ssment conditions for items meas- 
uring change identical from assessment to 
assessment so that any chants observed will 
be attributable to changes in achievement 
rather than a res^nse to an altered testing 
condition. Exercises that were packaged to- 
•gether in a booklet in the first assessment 
were packaged together in the second, and. 
they were sequenced in the same relative 
order in both assessments. However, in 
1972—73, mathematics and science were 
assessed together, while in 1977-78 each 
booklet assessing mathematics included only 
mathematics items. In addition, about half 
the 1972—73 items were released fo^^pubUca- 
tion following, the assessment and not re- 
assessed. Thus, althou^ exercises were in the 
same relative sequence for ^ch assessment, 
ekercis^ that separated som6 of the exercises 



in the first assasment were no longer present 
in the second. 

All exercise used to measure change were 
located tc^ether either at the beginning or the 
end of a package. Chanj^ in the scripts for 
the paced audio tap^ occurred between 
1972—73 and 1977-78;^ tapes for exercises 
measuring chan^ followed the 1972—73 
scripts. It was felt that switching between 
methods of reading exercises might prove 
confusing for students if items measuring 
change and new 1977—78 items were inter- 
mixed. 

Item booklets contained more items ahd/dr 
item parts in the 1977—78 assessment ^lan in 
the 1972—73 assessm^t. More multipart 
items and more muItiple<:hoice items, which 
generally required less time to administer, 
were included in the second assessment than 
the first. 



In 1972-73 the "t doot know" rtcpooM choict 
w«i read iloud for each sxareiM; for 1977—78 exer- 
diu that did not measure chii^e, it wu not. 
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CHAPTER 4 
SAMPUiNG 



This chapter describes proc^ui^s used in 
the sample design and sampie selection for the 
1972-73 and 1977- 78 assessments of math- 
ematics exercise. The sample design and 
sample selection for both assessments of 
mathematics were done by the staff of the 
Research Triangle InstiUite, Raleigh, North 
Carolina. ? 

The target populations for each of the 
assessments consisted of 9% 13- and 17-year- 
olds^ enrolled in eitlier public or private 
schools at the time of the assessment who 
were not functionally handicapped to the 
extent that they could not participate in an 
assessment. Specific groups excluded were: 
non-Ei^lish-speaking persons, those identified ^ 
as nonreaders, persons physically or mentally 
unable to respond, and persons in institutions 
or attending schools established for the phy- 
sically handicapped or mentally retarded. 

Nation^^' Ass^ment does not follow up 
specific individuals from one assesmaent to 
the next. In other words, the students who 
participated in the 1972—73 assessment are 
not the same ones who participated in 
1977—78. However, in each assessment year, 
part>i{«mts were carefully selected to re- 
present each age level For example. National 
Assessment assessed one set of probability 
sample of 9-year-olds in 1973 and a totally 
different set of probability samples of 9-year- 
olds in 1978« Each was a sample of the 



Dtfinitioo* of 1977—78 MitMmtnt af» groupt m\ 
9-y«»roIdi bwn duriag nksdsr y«tr 1968; 
13-yMr^dft — bom during cakncbr y«ar 1964; tnd 
l7-yMr-otdt - bora October i. 1960, Ihrou^ 
ScpUmbtr SO, 1961. 



population of students who were 9 years old 
during that assessment year. Thus, when we 
say that 9^year-olds' achievement declined 
between 1973 and 1978, we mean that 
students who were 9 years old in 1973 
correctly answered the ^uxie questions more 
often than those who we« 9 years old in 
1978. 

The definitions of the taiiget populations 
were identical in each assessment. However, 
the sample design used to obtain represen- 
tative samples of the target populations was 
modified somewhat between assessments. An 
overview of the general sample design 
approach is ^\«n below, foUowed by a 
description of differences between the 
i9t2~73 and 1977-78 procedures. 

Ovetvifiw of the Nation^ 
Asttsunent S»n|^e .Design 

National Ass^ment used a deeply 'strati- 
fied, three-stage national probability sample - 
design with owrsampHng of low-income and 
rural areas. In the first stage, the United 
States was divided into geogiai^ical units of 
counties or groups of contiguous counties 
meeting a minimum 'population size require- ' 
ment." These units, called primary sampling 
units (PSUs), w«re stratified liy region and 
size of community. From the list of PSUi, a 
sample of PSUs w,as drawn (without replace- 
ment) with probability proportional to popu- 
lation size m^usures, repi^senting all regions 
and si:^ of communities. Omsampliag of 
low-in€»me and exti^me-rural areas was first 
performed at this stage by adjusting thf> 
estimated population size meaMoet of those 
areas to incr^se sampling rates. Within PSUs, 
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Census Employment Survey Data were used 
to further delineate and oversample iow- 
income areas. Counties with hi^ proportions 
of rural families were also oversampied. 

In the second sta^e, ail public and private 
schbols within each PSU selected in the first 
stage were listed. Schools within each PSU 
were selected (without replacement) with 
probabilities, proportional to .the number of 
age-eligibles in the school 

The third stage of sampling occurred during 
'the data collection period. A list of all 
age-eligible students within each selected 
school was made. A simple random selection 
of eligible students, without replacement, was 
obtained, and item booklets were adminis- 
tered to selected students. Specially trained 
personnel selected the sample and adminis- 
tered the booklets. 

Each respondent in the sample did not have 
the same ^ probability of selection because 
some subpopulations were oversampled, and 
adjustments were made to compensate for 
some schools' refusal to participate and for 
student nonr^ponse. The selection prob- 
ability for each individual was computed, and 
its reciprocal was used to weight #ach 
response in any statistical calculation to com- 
pensate for unequal rates of ^sampling and to 
insure proper representation" in the popula- 
tion structure. 

The number of PSUs, schools vrithin PSUs 
and students within schools was determined 
by optimum samplinj* principle. That is, a 
sample design was selected that would achieve 
the maximum precision for a given level of 
resources. ' 

Table 2 displays the number of PSUs and 
schools within PSUs selected in 1972-73 and 
1977-78 by age. 



DiffiMWcss in Sampis Dflsisp: 
1972-73 and 1977-78 

The 1977—78 sample was drawn according 
to the following procedures. Two types of 



TABtE 2. Numbw of PSUi and Schools 
Within PSU S*t«ct»d in 
1972-73 and 1977-78 

1972-73 1977-78 
^Msnwnt AsscsMMflt 





No. of 


Mo. of 


No. of 


No. of 




PSUs 


ScTwois 


PSUs 


Schoiois 


AgeS 


116 


971 


78 


538 


Age 13 


116 


979 


78 


571 


Afie 17 


116 


798 


78 


540 



PSUs were identified: (1) la^e-size popula- 
tion areas deHned by the U.S. Bureau of the 
Census as Standard Metropolitan Statistical 
Areas (SMSAs) and (2) other contiguous 
non-SMSA counties grouped together to meet 
, certain minimum-size ^uiremrats. The first 
stratification of PSUs was by geograj^ic 
r^on, as defined by the Office of Business 
Economics, U.S. Department of Commerce. 
(See Appendix B, definition of regional sub- 
groups.) 

Within regions, I^Us were classified into 
five size-of-community (SCX!) cat^ories: 

sex: 1 PSUs corresponding to the 13 
largest SMSAs after adjusting the 
population size to compensate for 
ovetMmpKng low-income metropol- 
itan aieas« These l^Us haire selec- 
tion probabilities so large that, 
under our allocation procedures 
they are certain to be included in 
our sample each year. These PSUs 
jure designated as self-representing. 

SOC 2 PSUs corresponding to the reinain- 
ing 57 SMSAs. with over 500,000 
populatic^. 

,?> SOC 3 PSUs corresponding to the remain- 
ing 162 SMSAs, 

SOCf4 PSUs made up of ||on-SMSA 
and 5 counties. SOCs 4 and 5 are deter- 
minod %q that half of the remaining 
population (aft^ adjustment for 
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oversampUnjj of rural areas) falls 
into each category. SOC 4 contains 
PSUs in which less than 60% of the 
r^idents are classified as rural. 

The self-representing PSUs reprint ad- 
ditional stratiHcation, making an effective 
total of 17 (13 * 4) size-of-community strata. 
Each self-representing SMSA was divided fur- 
ther into geographical substrata or non- 
overlapping replicates that constitute multi- 
pies of convenient work units for item 
administration. These multiple work units 
were included with the rest of the nonself- 
representing PSUs to form the pool from 
which first-stage sampling units were selected. 
To insure adequate repreu»ntation, National 
Assessment doubled the sampling rate of 
low-income and rural areas. 

In 1975—76, first-stage units were selected 
simultaneously for four consecutive assess- 
ment years (1975-76 through 1978-79), as 
were schools in the self-representing PSUs. 
The sample design required that every four 
years we assess at least once in every state and 
not more than once in any school. There are 
1,101 primary sampling units in the primary 
sampling frame for the four-year period, from 
which about 75 first-stage sampling units are 
selected each year. 

Within the primary strata, public and pri- 
vate schools were listed and further stratified 
by the estimated number of youngsters eligi- 
ble at each age. Small schools wete clustered 
until they were large enou^ to respond to 
the same number of packages as the larger 
schools in a . stratum. Schools or school 
clusters were selected without replacement, 
with probability proportional to the number 
of age-eiigibles in the school or cluster of 
schools. Once schools were identified, dis- 
tricts were contacted to check for changes in 
grade range and for the existence of new 
schools. This information was used to revise 
probabilities of schools' selection. 

In the third stage, students were, selected 
with equal proi»bility and without replace- 
ment within each sample school. The number 
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of students selected is proportional to the 
number of age^ligifaies, with oversampling in 
low-income and rural areas. 

During data collection, allowing for vari- 
able group sizes for each' package adminis- 
tration within schools enabled National 
Assessment to obtain desired sample sizes in 
schools having characteristically lew response 
rates. This feature also permitted last-minute 
modifications and adjustWnts to selection 
probabilities necessitated >by enrollment 
change. 

The sampling procedures used in 1972—73 
differed somewhat from those us*>d in the 
1977—78 assessment (Final Report 
Sampling and Weighting Activities, . . . 1979; 
Moore et al., 1974). In 1972-73, the PSUs 
were stratified by region, size of community 
and socioeconomic status (SES). In addition, 
the sample was constrained to include all 
states. The sampling of PSUs in 1972-73 was 
accomplished by using a controlled selection 
procedure (Moore et al., 1974). In 1977-78, 
PSUs were stratified by region and size of 
community, with the constraint that each 
state must appear in the sample once every 
four years. Controlled selection of PSUs was 
no longer required. 

The 8iz©K)f-community stxatifScations in ' 
1972-73 were different from those in 
1977—78, which were described earlier in this 
chapter. There were only four SOC strati- 
fications in the first assessment of math- 
ematics. The first SOC category in 1972-73 ' 
consisted of all central cities with overall 
population "greater than 180,000. The second 
SOC cat^ory Consisted of the remainder of 
the SMSA containing the central city in SOC 
1. The SOC 3 category in 1972-73 consisted 
of the remainder of the SMSAs and all 
counties not in SOCs 1 and 2, containing at 
least one city with a minimum population of 
25,000. In 1972-73, the SOC 4 category 
consisted of all the remaining counties not in 
SOCs 1, 2 or 3. 

In 1977—78, ovemmpling of low-income 
metropolitan areas and extzeme-ruxal az«as 
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was uccompli&hed at the primary stage by 
increasing the estimated population size meas* 
ures of PSUs containing these areas and then 
sampling with probabilities proportional to 
those adjusted size measures- In 1972-73, a 



poverty index was used to stratify PSUs into 
high- and low^ES stratifications. The 
sampling rates within these strata ware then 
increased in order to achieve the desired 
oversampling* 



14 ■ 



CHAPTER 5 
DATA COLLECTiOIVS 



Professional dat^a collection staff from the 
Research Triangle Institfite, Raleigh, North 
Carolina, administered the assessment 
booklets. This staff was used to minimize the 
burden on participating schools and to ensure, 
insofar as possible, uniform administration 
conditions across the country (Final Report 
. . . In-School Field Opemtions..., 197S), 

Participation in the National Assessment is 
vpluntarsR. NAEP makes every effort to 
encourage the schools selected in the sample 
to participate in ti. .issessment, and National 
Assessment and jsearch Triangle Institute 
staff have obtaii. high rates of school 
cooperation, as she a in Table 3. Student 
cooperation rates were also high, especially 
for 9- and 13-year.oId^ {Final Report 
Field Operations..., 1978, p. 35, Table 22). 
The effect of student nonresponse is discussed 
in Appendix D. Table 4 shows the actual 
number of students that responded to a 
particular exercise booklet at each age level in 
the 1977-78 assessment. 

Since National Assessment reports data for 
groups of students, not individuals, it is not ' 



necessary for each' student to respond to 
every item. In 1877-78, booklets were 
administered to groups of 10-25 students; 
each poUp responded to only one booklet.' 
The groups varied in size depending on an 
estimate of the rate of nonresponse for a 
particular school. In 1972-73, the planned 
session sizes were fixed at 12 students at each 
age. 

National Assessment takes steps to 
guarantee the anonymity of each respondent. 
Students' names were listed with their 
booklet identification number so that scoring 
and prot^ng personnel could go b^ck to the 
school lists for data verification — for 
instance, on background information - if 
necessary. These lists did not leave the schools 
and were destroyed six months following the 
ass^sment in a ^ool. , 

School officials were asked to respond to a 
•'principal's questionnaire." This 
questionnaire asked about the enrollment in 
various grades, the types of commimities in 
which the students lived and the general 
occupational levels of the people in the 
community. Principals of S-yiiar-oids' schools 



TABLE 3. School CoofMration Ratn, 1977^78 Amtsmtnt 



Agt 



9 

t3 
17 

OwaU 



Ptorotnt of EiigibI* Sdtoofi 
Pw^p«inf in 1977-78 
Asunmmt 

9t.5 
91.3 

m.s 

90.8 
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TABLE 4. NumiMr of Studtnts Rfssxsnding to 
Eadi itim Bookitt in 1977-78 AsHumant. by Agt 





A«aS 




Ag«t3 




Age 17 


Booklet 


Number 


Bookfai 


Number 


Bookitt 


Number 




R«i pending 




Rtsponding 




RMponding 


1 


1,496 


1 


2,423 


1 


2,295 


2 


2.463 


2" 


2.433 


2 


2,274 


3 


2,438 


3 


2.469 


3 


2,273 


4 


2,445 


A 




A 
H 


t ICC 


S 


2^429 


s 


2.462 


5 


2.264 


6 


2,482 


6 


2.440 


6 


2.233 


7 


2.437 


7 


2.316 


7 


2,254 






8 


2.431 


8 


2,221 






9 


2.368 


S 


2,238 






10 


2,431 


10 


2,216 






11 


2.4S1 


11 


2,199 










12 


2.124 


Total 


17.190 




26.661 




26,756 



were asked to indicate the number of minute 
per week devoted to mathematics instruction 
and whether the time devoted to mathematics 
computation had increased, decreased or 
remained the same over the past five years. 
Examples of forms used to collect 
background information &om students and 
school officials are located in Appendix C. 

The assessment administrator coded each 
student's birth date, sex, grade, racial/ethnic 
cslassification and identification number on his 
or her booklet. Administrators made a visual 
racial/ethnic identification at the time each 
student tum^ in his or her booklet. During 
the 19t 7-78 assessment, six different racial 
classifications were used: white, black, 
Spanish heritage, American Indian or Alaskan 
native. Pacific Islander or Asian, and 
unclassified. If an administrator was unsure of 
a student's racial/ethnic group, he or she 
referred to the student's name or listened to 
the student talk to make the identification. 
Students were not verbally asked to give a 
racial identification for themselves by the 
assessment administrator; however, 
17-year-old students were asked to provide 
this information in one of the background 
questions included in the exercise booklet. 



Sample sizes of the two classifications 
American Indian or Alaska:, "stive and Pacific 
Islander or Asian are too sinall to pennit 
reporting for these ^oujb. Results for the 
group classified as Spanish heritafe cannot be 
reported for separate exercises but can be 
reported for aggregate results across a number 
of exercise. Change results between 1972—73 
and 1977—78 are reported for whites, blacks, 
and Hispanos or Spanish sunuuned. 

Each age group was assessed in 
approximately the same time of the school 
year in each assessment. As noted previously, 
13-year-olds were assessed in 
October— Decern ber, 9-year-oids in 
January— February and 17-year-olds in 
March— May. 

Following data collection, assesanent 
administratont sent completed booklets to the 
scoring contnu;tor, Measiiiem«it Research 
Center, Iowa City, Iowa.* Booklets were 
quality-checked to verify that correct 
administrative procedures were being 
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followed fay the field staff and that ail 
booklets were accounted for. Coded 
kientification information was also checked 
for accuracy; inconsistencies that couid not 
be reconciled were sent back to the 
assessment administrator to be checked 
against the list of student names and 
identification numbers retained by the school 
for six months following the assessment. 



• In 1972—73, IZ-year-oIds who were not 
currently attending school and young adults 
aged 26^35 were included in the mathematics 



assessment. Out-of-schooi 17-year-olds and 
adults could each answer up to four booklets 
of assessment materials; they were paid $5.00 
for each booklet that they completed. 
Unpaced audio tapes were used for these 
groups. Assessment of out-of-school 
17-year-olds and adults is quite expensive, and 
in 1977—78, funds to assess these groups were 
not available. Since out-of -school 17-year-olds 
were not assessed in 1977—78, only insults 
for 17-year-olds attending .school in 1972—73 
and 17-year-olds attending school in 1977— 78 
were used in calculating changes in 
17-year-olds^ performance. 
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CHAPTER 6 
SCORING 



The data that National Assessment collects 
in the field must be converted into a form 
suitable for computer processing and analysis. 
This conversion was done by the 
Measurement Res^di Center. Iowa City, 
lowa,^ the contractor in charfi^ of printing 
the asses^ent item booklets, receiving, 
scoring and,machine-pr<K3essing the data. 

In the 1977—78 assessment, ^proximately 
half the items were multipie-choice and half 
were open-ended. Responses to 
multiple-choice exercises were i^ad directly 
by optical scanning machines. The scoring 
contractor employed^ special staff to hand 
score open-ended exercises. Scorers were 
responsible ifor cat^orizing responses, using 
the scoring guides for open-ended items that 
defined categories of acceptable and 
unacceptable responses, and were responsible 
for coding the category of responses for an 
item into ovals that could be read by the 
optica! scanning machine. 

Scoring guides for open-ended exercises 
were developed using field bryout data. 
Scoring, categories included likely errors and 
thus were useful in identifying frequently 
made mistakes. 

For chan{^ in performance to be measured 
accurately, scoring had to be the same in each 
assessment year. For multiple<hoice items, 
the tame responses were scored' correct in 
each year. Scoring guides for op^-ended 
items used to measure change we» 
revised in ld77-78 to be consistent with the 



Now W«itiiifhottit DsUScort Systems, .lofwi City, 
Iowa. 



guides for items first administered in 
1977—78. All responses to open-ended items 
made in 1972-73 were rescored at the same 
that the 1977—78 responses were scored, 
usilife the revise scoring guides, to ensure that 
^ring of the two sets of data was identical. 

Scorerji were trained in the use of the 
scoring guides by ^scoring sample responses 
taken from arriving assessment data; National 
Ass^ment staff was involved in the training 
process. Scorers initially worked as a group 
and disclosed the appropriate categorization 
of each ejjample response. They then worked 
individually on another set of responses. 
Discrepancies were resolved and explained. 
Once the group felt comfortable using the 
guides, they started categorizing the actual 
data. Supervisory personnel checked all work 
done for the first few days of a scoring effort 
to be sure that scoring was consistent. 

To further ensure the quality and 
consistency of scoring open-ended exercises, 
quality-contojl chec^ks were conducted during 
the sa>ring of these exercises. At regular 
intend, randomly selected responses were 
drawn from the total pool of responses for an 
item and read by randomly selected sc»r^. 
Both the r^ponses and the scorers were 
elected without replacement; approximately 
iO% of the responses were included in the 
quality-control check. Score* for the 
quality-control reidingB were i«corded, and 
the responses selected for quality control 
were then put bick into the total pool of 
responses to b6 scored during the regular 
course of scaring. Following scoring of aH 
mponses, the" two scores for quality-control 
renxHises were compared. It discrepand^ ir« 
scoring became apparent, scorers were 



retrained and, on some occasions, work was 
rescored/ 

Percentages of agreement between 
quality-control and regular scoring were 
computed for each open-ended exercise. 
These data are summarized in Tabie 5. There * 
is a greater range and a slightly lower average 
, t>ereent of agreement as age increase. Since 
the age 13 responses were scored first, the age 
9 responses second and the age 17 responses 
last, the decline in average percent of 
agreement and the increase in range were not 
solely due to ••staleness*' of the scorers over 
time. A more likely hS^pothesis is that the 
exercises (thus, their associated scoring 
guides) became increasingly more complex 
and difficult for the older age levels. 

Scoring for each age group tegan during 
the administration of the assessment to th&t * 
age group. Scoring for each age group took ^ 
from 6 to 12 weeks. Scorers were divided into 
two teams, and each team learned th? scoring 
guides and scored about half the o|^n^nded 



TABLE 5, Quality Cwitrcrf Summaries for the 
, 1977-78 Mathematics Assessment 
Open-Ended Exercises, by Age 

Age 9 Age 13 Age 17 

Total number of 
responses 

sampled 70,947 84,470 1 06,732 

A. 

Range of average 
percent of score 
agreement for ^ 

exercises 94-100 92.3-100 86.4-100 

Average percent 
,of $OTre agree- 
ment across * 

exercises 98.9 98.5 97,7 




items, at each age level Approximately 10 to 
20 scoreiB were working on scoring the 
mathematics data at any given time. 
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CHAPTER 7 
DATA ANALYSIS 



Measures of Ai^ievement 

As stated previousiy. National Assessment 
reports the performance of groups, of 
students, not individuals. The basic measure * 
of achievement reported by the assessment is 
the percentage responding acceptably to a 
given item. This percentage is an estimate of 
the percentage of 9-, 13- or i7>year-olds who 
would respond acceptably to a given item if 
every 9-, 13- or 17-yeai-old in the country 
were assessed. 

Percentages of correct responses are used 
because each item is designed as a separate- 
measure of some aspect of an objective or 
subobjective. T4ie purpose of National 
Assessment is to discover if more or fewer 
people are able to answer these items 
correctly — and thus meet »,he objectives - 
over time. Some items - such as attitudinal 
questions - did not have "correct" answers. 
For these items. National Assessment 
reported the percentages responding in 
various ways. 

In addition to providing national results, 
National Assessment reports on the 
achievement of various subpopulatiohs of 
interest. Groins are defmed by region of the 
country, sex, race, size and type of 
community lived in and level of parents' 
education. For the 1977-78 mathematics 
assessment, results were also analyzed by 
different levels of mathematics course-taking 
reported by 17-ye&r-oIds. Results for. some 
additional variables were also analyzed. 
Infc^ation on these variables is found in 
Af at hematics Technical Report: 'Summary 
Voiume (1976). Definitions of the reporting 



groups ar^ found in Appendix B. ' 

Procedures for estimating percent^s of 
acceptable responses to exercises are 
dependent on the sample design. Each 
response -by an individual V' weighted and 
multiplied by an adjustment factor for 
nonresponse.' An estimate of the percentage 
of a particular age group that would have 
responded to an exercise acceptably if the 
entire age group were assessed is defined as 
the weighted Tiumber of acceptable responses 
divided by the weighted number of ail the 
responses. A similar ratio of weights is used fo 
estimate percentages of acceptable responses 
for reporting groups or subpopulations of 
interest.'^ 

The difference between the percentage of 
acceptable responses for a reporting group 
and that of the entire age group (nation.) on 
an exercise describes the performance of any 
Reporting group relative to the entire age 
group. This difference is a p^itive number if 
the group achieves a hi^er percentage than 
the entire age group and is a negative number 
if the group achieves, a loWer percentage. For 
example, a group performance of +i g 
mdicates that the percentage of acceptable 
r^ponses for th§.. group is 1.8 percentage 
points higher than the national percentage of 



'Appendix D discusws nonresponie in sueument 
umpief. 

^Following the 1976-77 aMestnwnt, a weighting- 
cUm adjustment prowdure was used to amooth e«- 
Um«t«d population i»-oportic»is acroct the eight 
aisestmentt conducted between 1969—70 and 
1976-77. DocuraenUtion of this procedure and m- 
ii mated population proportions am included in 
Appendix F. 
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acceptable responses for a particular age level. 

Increases or decreases in the percentage of 
acceptable responses between Xwo ass^sments 
are estimated by finding the difference 
between percentages obtained from each 
assessn^ent, A positive difference indicates an 
increase, and a negative difference indicates a 
dei'rease in the {>ercentage of students ,who 
responded m a particular way from one 
assessment to the^next. These differences, or 
change measures, are used to indicate trends 
in achievement, or performanc*e, for an age 
level or subpopulation of interest. Changes in 
group differences from the national 
performance between two assessments are 
Msed to indicate the relative trend of a group 
cc^mpared with the national trend of the age 
group. 3 

To present a jjeneral picture of changes in 
achievement. National Assessment 
summarizes the gains or losses on each 
-exercise (either for the entire learning area or 
for some integral set of exercises) by using the 
mean, or arithmetic average, of the changes in 
f>erientages of acieptabie responses to the 
exercises. During the first years of National 
Assessment, the median was used as the 
principal summary . measure. However, the 
mean was chosen as the principal summ^ 
measure of change after extensive 
investigation showed empirically that it was 
more suitable Tor National Assessment change 
data than alternative measures.^ In addition, 
the mean is an easily understood and. well- 
khown statistic and has simple arithmetic pro- 
pertw>yseful for the analysis of differences or 
chafige measures — in particular, the differ- ' 
ence between means is the same as the mean 



d i f f eren ce . This property allows us to 
accurately describe the mean cliange as the 
difference between mean percentages of ac- 
ceptable response from one assessment to the 
next. Mean percentages ar« used' to simplify 
dMcriptions' of change and 'd^cyiptions of dif- 
ference; in group performances. Differences 
in mean perform^ce between reporting 
groups and the nation are used to summarize 
the performance of various groups relative to 
the nation. We do not intend that the mean 
percent^e should iie construed jas an average 
test^ore. ^ ' ' 

Unless the items summarized in the mean 
percentages of acceptable response are iden- 
tical, the means of one age gtbup should not 
be compared to the means of another, since 
their values reflect the choice of exercises in 
addition to the performance of the students. 
When only a few exercises are summarized by 
a mean, we should be specially cautious in 
interpreting results, since a small set of e?:er- 
cises mi^t not adequately cover the wide 
range of potential behavior^ included under a 
given objective or subobjective. The mean 
should be interpreted literally as the arith- 
metic average of the percentage of acceptable 
responses obtained .from National Assessment 
samples on a specific set of exerdses. 

In considering National Ass^ment's 
achievement measure, it is the differences in 
perfofmance betw^n assessment years, a- 
moni^ groups and among ages that. are the 
most useful. By maintaining the same item or 
set of items in making these comparisons, we 
have a reasonable indicator of whether more 
or fewer people know or can do something 
judged important 



Twenty-two empiric^ distributions of chftng« 
mwures from the 1969^70 and 1972~7S science 
^Uf^menis were used to generate Monte Carlo »i- 
mulitsons of umpiing distributions for several mesi- 
ures of centrs! location. In addition to the mean and 
median, other measures &f centnl location that were 
considered in the simutatlon studies induded^the 
average of tbe extremes, two fomu of biwrighted es- 



timates and three fomas of weight^matching estima- 
tors described by John W. Tukey in the research 
paper ''Some Ck>nd^ratioiii on Locatovk Apt for 
Some Sqi^ezed'Tai! (and Stwtcbed^TaU) Parents** 
.(1975). In almost every case, the sampling sta&iHty of 
the mean change was as good as or better than that of 
the olhi^ meames studied 
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Estimating Vari^iiity in 
Ai^ievemant Measures 

National Assessment uses 'a national prob- 
ability sample at each age ievel to estimate the 
proportion of people who would sUccessfuliy 
complete an exercise. The particular sample ' 
selected is one of a large number of ail pos- 
sible samples of the same »ze that could have 
been selevted with the silme sample design. 
Since an achievement measure computed 
from each of the possible samples would dif- 
fer from one ;sample to another, the standard 
error- of this statistic is used as a measure of 
the sampling variability among achievement 
measures from all possible samples. A stan- 
dard error, based on one particular sample, 
selves to estimate that sampling variability. 

In the interest of sampling and cost effi- 
ciencies, National Assessment uses a complex^ 
stratified, multistage probability sample de- 
sign. Typically, complex designs do not pro- 
vide for unbiased or simple computation of 
sampling errors. A reasonably good approx-' 
imatjon of standard error estimates of suicept- 
able response percentages is obtained by 
applying the jackknifd procedure (Miller, 
1964, pp. 1594-1705; Mill^, 1968; pp. 
567-82; MosteUer, 1968) to first-stage 
sampling units within strata. Standard errors 
for achievement measures such as group dif- 
ferences, mean percentages or mean group dif- 
ferences for a particular assessment year are 
estimated directly, taking advantage t)f fea- 
tures of the jackknife procediire that ^ gene- 
ric to all of these statistics,* Since saihples for 
different assessments are independent, the 
standard errors of the differences in achieve- 
ment measures^ between as^sments can be es- 
timated simply by the square root of the sum 
of squared standaiti errors from each of the 
assessments. 

The stands^ error provides an estimate of 
sampling reliability tor the achievement meas- 



Sm Appendix E f or « nao» det^ed des<^pUon of 
Nition»l AjwMwnent't computation of standard 
•rtoa. ^ 



ures used by National Assessment. It is com- 
prised of sampling error and other random 
error associated with the assessment of a spe- 
cific item or set of items. Random error in- 
cludes all possible nonsysteraatic error asso- 
ciated with administering specific exercises to 
. specific students in specific situations. Ran- 
dom differences among scorers -for open- 
ended items are also included in the standard 
errors. 

Differences in performance between assess- 
ment years or oetween a reporting group and 
the nation are reported as differences only if 
the differences are at least twice as la^e as 
their standard errors. Differences this large 
would occur by chance in fewer than 5% of 
all possible replications of our sampling and 
data collection procedures for any particular 
reporting group or national estimates. . 

Controliing Nonr^dom Errors 

Systematic errors can be introduced at any 
stage of an assessment — exercise develop- 
ment, preparation of exercise li(W)klets, design 
QT- administration procedures, field' adminis- 
tration, scoring or ^analysis. These non- 
sampling, nonrandom errors rarely can be 
quantified, nor can the magnitude of the bias 
they introduce into our estimates be evalu- 
ated directly. 

Systematic errors can be contro^ed in la^e 
part by employing uniform adminis&atioh 
and scoring procedures and by requiring rig- 
orous quality control in all phases of an 
- assessment. If the systematic errors are the 
same from age to age or group to group, then 
the differences in percentages or mean per- 
centages are measured with reduced bias be- 
cause subtraction tends to cancel the effect of 
the systematic errors. 

Similarly, the effect of systematic errors in 
different ass^sment years can be cmitroll^ 
by mefully replicatittg in the second assesti- 
ment the" proi^dtir^ carried out in the fiwt. 
' Differences in adiievement acrc^ assessment . 
yean will also be measures with reduced bias 
since subtraction will again tend to amtiel 



'Systematic* errors. . . years apart. National Assessment has made 

every effort to keep C9nditipn$ as nearly the 
■ Although it is not possible for every con- same as possible. Changes in procedures de- 
dition or procedure ta remain exactly the , . scribed in this report were judged to have a 
same bet\yeen assessments conducted several jreiatively minor impact. 
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CHAPTERS 

REPORTS ABOUT THE MATHEMATiGS ASSESSMENTS 



Each assessment generates a tremendous 
amount pf'data. To make these data as useful ^ 
as possible to a variety of audiences. National 
Assessment provides several typ^ of data re- 
ports. 



Summary Reports 

Since it is difficult and time consuming to 
synthesize many discrete bits of data. Nation- 
al Assessment prepares reports fo!*the general 
- public - including parents, classroom teach- 
e«?, school administx&to» and legislators — 
that summarize and hii^Ught assessment re- 
sults. Summary reports for the mathematics 
assessment are organized by cognitive-process 
level. 

Although National Assessment does not in- 
terpret 'assessment results, it recognizes that" 
data presented alone are often difficult to 
consider in perspective. Accordingly, NAEP 
asked a group of mathematics educators to 
review and comment upon the results; their 
comments are included in the reports! 

The following reports, desOTbing. 1977—7$ 
status in mathematics achievement and 
changes between 1372-73 and 1977— 78, are 
available: Changes in Mathematical Achieve- 
ment; 2973-78 (1919); Mathematical Knowl- 
edge and Skills (1979); Mathematical 
Applications (1979); and Mathematical Un- 
derstanding (1919). 

ObjMtivfls 

A description of the 1977—78 mathematics 
objectives and the pcoceduxet used in develop- 



ing the objectives and items for the 1977—78 
assKsmeat are available in: Mathematics Ob- 
jectives, Second Assessment (1978). 

Exercise Lev^ and Summ^^Oata 

For those wishing to use specific National 
Assessment items, NAEP provides copies of 
released items, ex^cise documentation (in- 
cluding exercise timing, objecUve being 
measur«i, and so forth) and exercise-level 
data. The materials available axe: 

The Second Assessment of Math- 
ematics, 2977-78, Released Exer- 
cise Set (1979) - this loose-leaf set 
provides copies of all released math- 
ematics .exercises from the 
1977—78 mathematics assessment, 
exercise documentation (including 
timing, objective measured and 
exercise administration mode) and 
• percentages of correct r^pQnsi*s for 
the nation, regional grom», males 
and females and coaamunity-siae 
groups. 

Data Appendix for the 2977-78 
Mathematics Relegesed Exercise Set 
(1980) — this microfiched appendix 
provides additional data on the 
exercises included in the Released 
Exercise Set. It presents per- 
centages of respondents giving each 
resppn*e alternative for an exercise 
at ^an age level for the nation and 
for; 10 reporting variable: region, 
sex, race/ethaicity, level of pamital 
education, tyi» of community, dze 
of community, grade level, modal' 
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^de by region, modal grade by 
community size; and modal grade 
by sex. . * 

In addition to exercise-Ievei data. National 
Assessment computes means for a learning 
area and for selected sets of exercises within 
the learning area. In mathematics, for 
example, means were computed for the var- 
ious cojaiitive-proceSs levels and content 
areas. These means are useful in. comparing 
performance between assessment years and 
•l>erween repqrting groups. Means art com- 
puted for a number of reporting variables, in- 
cluding region, sex, race/ethnicity, size and 
type of .community and level of parental edu- 
cation, as well as additional variables on 
couree-taking for IT-year-olds. These sum- 
mary data are found in: Mathematics Tech- 
nical Ref}ort: Summary Volume (forth- 
coming). 



Fai>ik:UifrQ«tarape 

For those who wish to perform their own 
analyses of National Assessment data, NAEP 
makes available a data tape of 
respondent-level data for the X9t7-78 math- 
ematics assessment. To protect the confiden- 
tiality of the respondents, all Identifying in- 
formation (school, district, state) has been 
deleted. The tape also includes documen- 
tation of exercises and classification schemes 
used by NAEP to summarize exercises. The 
tape is organized and documented to be easy 
to use with standard statistic^ packages. 

Us^ Services 

National Ass^ment provides some assist- 
ance to those wishing to um ass^ment items 
or to repiiraite assessment methodplc^. 
Those interested in jreceiving assistance should 
contact the National Assessment office. 
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APPENDJX A 

MEMBERS OF THE MATHEMATICS ADVISORY BOARD 



Sarah T. Herriot, Henry M. Guhn Sr. High School, Paio Alto, Calif. 

Prince Jackson, Savannah State Cbllege, Savannah, Ga. 

Jeremy Kilpatnck, University of Georgia, Athens, Ga. 

Donald L. Kreider, D^mouth College, Hanover, N.H. 

John LeBianc, Indiana Univereity, Bl<K>mington, Ind. 

James W. Wilson, University of Georgia, Athens, Ga. 
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.APPENDIX B 

DEFINITIONS OF AiATIONAL ASSESSMENT REPORTING GROUPS 



^ In addition to reporting, r^ults for ail 9-, 
13- and 17-year-oId students in the United 
States, National Assessment rep<tfts results for 
a number of population subgrou{«. Mc»t of 
these suisgroups itre defined for both the 
1972-73 and the 1977-78 mathematics 
assessments. Data on perfonnan<% for dif- 
f^nt levels of course-taking by 17-'yea^-oIds 
are only available tor the 1977-78 assess- 
ment. 

Definitions of the subgroups follow: 



Rigion 

The coimtry has been divided into four re- 
gions; Northeast, Southeast, Central and 
West. States included in each re^on are 
showm on the following map. 



Raci 




Results are nresented for blacks, whites and 
Hispanos.^ 



Uvti of pWmtai £(&i»ti<» ' ' 

Three categoric of parental>educatioq lev- 
els are defmed by National Ass^sment, based 
on students* reports. Th^ eateries &x«: (1) 
those whose.i parents did not graduate from 
' high school, (2) those viho have at le&it pne 
parent who paduated from high school and 
(3) those who have at least one parent who 
has had some post-hi|^-school education. , 



Typt of Commtmity 

Communities in this <»tegory ere defined 
by an occujj^tionai profile of the area served 
by a sdiool as well as by the size of the com- 
munity in whidi the school it located. 

Advanta^d-urban communitie$. Students 
in tihii group attend schools in or around 
cities having a 'population gteater than 
200,0CXJ where a high proportion of tho res- 
ictents are in pro£esiional or maoag^a! posi' 
tions. 

HiiBadvantaged'Urbstn i^mmuniti§i Stu- 
dmsts in this group attend schools in ox 
around dtifis having « p^uSation.greatef timn 
200,000 where a selativ^ high proportion of 



R«iults are i^rted for males and femates. 



Otte for HiQiiisQS sm reported only for wto of 
M«ciMi and not for imiividuAl item* Kf^t ^ft of 
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the residents are on welfare or aie not r^iiiar- 
ly employed. 

Extreme-rural communities. Students in 
this group attend schools in areas with a 
population under 10.000 where many of the 
residents are farmers or. farm workers. 



Si2t of Community 

Rig cities. Students in this group attend 
schools within the city limits of cities having a 
1970 census population over 200,000. 

Fringes around big cities. Students in this 
group attend schools within metropolitan 
areas (1970 U.S. Bureau of the Census ur- 
banized areas) served by cities having a popu- 
lation greater than 200,000 but outside the 
city limits. 

Medium cities. Students in this group 
attend schools in cities having a population 
between 25,000 and" 200,000, not classified in 
the fring^-around-big-cities category. 

Small places. Students in this group attend 
schools in communities having a population 
less than 25,000, not classified in the fringes- 
around- big-cities category. 



Gradt in School 

Results are categorized for 9-year-olds in 
the 3rd or 4th ©rade, 13-year-oIds in the 7th 
or 8th grade, and 17-year-olds in the 10th, 
11th or 12th grade. 

Model Gradt by Ragion 

Results are categorized for 13- and 



17-year-old respondents in £^es 4, 8 and 11, 
respectively, who live in the Northeastern, 
Southeastern, Central or Western regions of 
the country. 



Mod^ CSrada by Communfty Siza 

Results are categorized for 9-, 13- and 
17-year-old respondents in grad^ 4, 8 and 11, 
respectively, who live in big cities, fringes 
around big cil^i^, medium dties and small 
places. 



Modai Grade by Sex 

^ ' /* 
Results are categorized for 9-, 13- and 
17-year-old males and female in grades 4, 8 
and 11, respectively. 



Mi^matid Courm Talun 
by 17.YMf^Oidf 

Seventeen-year-olds were asked about their 
raathentiaUcs course-taking experience. A stu- 
dent was said to have tak^ a course if he or 
she had taken it for one-half year or longer. 
Five levels of mathematacs-couise experi^ce 
are defined: (1) those who had not taken Al- 
gebra or any more advanced mathematics 
course; (2) those who had taken Algebra I 
but no more advanced courses; (3) those who 
had taken Geometry but not A^ebra 11 or 
any more advanced courses; (4) thoie who 
had Uken Algsfara 11 but no more advah^ 
courses (Trigonometry, Probability and Sta- 
tistic, or Pre4:alcului|; and (5) those who 
had taken Algebt-a II and at leaat one more 
advanced course (Trigonometry, Probability, 
and Statistics, or Pre>Calculus). 
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APPENDIX C 

FORMS USED TO OBTAIN BACKGROUND INFORMATION 

This appendix includes the forms used by spondents for the 1977-78 a^ssment Fol- 

National Assessment to collect background in- lowing are a listing and a brieTSJLorof 

formation from school officials and re- the forms included ^ "^^^i aescnpUon of 

pp. 32-33 School Principars Questionnaire - filled out by school principals or other 
school Officii ror «..hools « each of the age level. discu««i. 

pp. 34 -35 Principals Mathematics Questionnaire - given only to ejpmentary school 
. . principals. Provides information about the elementary school's mathematics 
program. 

pp. 36-37 Standard Background Information Form for B-Ycar-Olds - provides in- 
formation about reading material in the home and level of parents' edu- 
cation. ' 

m 

pp. 38-39 Standard Background Information Form for IS-Year-Olds - provides in- 
formation about reading material in the home, lev^l of parents' education 
' and place lived in at age 9. 

pp. AO- .13 Standard Background Information Form for 17-Year-Olds - provides in- 
formation on homework, TV watching, racial identification, possessions in 
the home and classroom activities, in addition to questions also asked of 9- 
and i 3-year-olds. 

P- 44 ^<^^iind Information on Mathematics Experiences for 9'Year-OIds - pro- 
vides data on metric ^d hand calculator experiences. 

P. 45 Background Information on Mathematics Experiences for 13- Year Olds ~ 
provides data on metric and hand calculator experiences, 

pp. 46-47 Background Information on Mathematics Experiences for 17-Year-Olds — 
provides data on metric, hand calculator and mathematics course-takiniz ex- 
periences. ' • 
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Schooi Principars C^estionnalre 



Thic npdft ■utrHsrinS by law <20 U.S.C. t22t e<9l. Whtti 
trois Art not raqukid to ftipcM^d, your coopMtion it rmmM 
to tfi« rtniltt of ^ iuffv«v compri^tM^ aocurf^ 



Primary Saapling Unit 



PLEASE 
PRINT 



Naoe ot School 



Address of School 



School Nuxaber 
Age Group (s) 





1 













(Street) 



(City) 

Natoe of School Principal 



(St^te) 



13 



17 



(Zip Code) 



Naoe and title of person completing the form if other thaa school principal 
Naae Title 



What is your best estimate of the current enrollment and the average daily 
attendance by grade of your school (1977-78 school year)? (Enter zeros for 
grades not served by your school.) 



Grade 


K ' 


1 


2 


!■ 


4 


5 


6 


7 


8 - 


9 


10 


11 - 


12 


EnrollacQt 




























Average 
Daily 

Attendance 
























« 





2. Approxifljately what percentage of the students attending your school live in each 
of the following areas? 

t A In a rural area (less than 2,500) 



_Z B In a town of 2^500 to 10^000 



XOCZ 



X C In • town of 10,000 or iiore 

(Ites2s A-C should add to lOOZ) 
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3. Approximately what percentage of the students attending your school are children 

^ A Professional or nsanageriai personnel 

B Sales, clerical, technical or skilled workers 
^ C Factory or other blue collar workers , 
S D Fans workers 



JS E Persons not regularly employed 



100% 



% F Persons on welfare 

(Items A-F should add to 1002) 



4* Approximately wha;: percentage of the students attending your school arc 

2 A American Indian or Alaskan Native 

2 B Asian or Pacific IslaiK*,cr 



X C Hispanic, regardless of race (Mexican, Puerto Rican, Cuban, Central 

or South American or other Spanish culture or origin) 

JE D Black and not Hispanic 

i E White and not Hispanic 

(Items A-E should add to 100%) 

lOOZ J 

5. Does your school qualify for ESEA Title I assistance? 

Yes - If Yes, approximately what number of students qualify for 

Md whrtC Dyabei of &tu4er.ts are receiving ESEA Title I assistance? 

- Approximate number of students qualifying for ESEA 

Ti$^le I assistance 

_ Approximate number of studcDs receiving ESEA Title I 

.. assistance. 

No 



THANK YOU FOR YOUR COOPERATION 
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Prindpai'c Mathsmaties Questionnaira 



The purpose of this questionnaire is to provide additional information which will be 
used in the analysis of the mathematics data. Please answer the following questions 
for each school ^ade listed. Darken the appropriate ovals with a soft lead pencil. If 
you have questions about any of the following items, please contact the National 
Assessment District Supervisor. Thank you for your cooperation. 



1. For each srrade level in your school, how many minutes per week , on the 
average, does each student spend in a specifically scheduled mathematics 
class? 



Minutes Per Week 



garten 


level 
CD 


0 
CD 


1-50 


51100 
CZ5 


101-150 


151-200 

t ^ 


201-250 


251-300 
CD 


300 or 
more 
t ^ 


B. Grade 1 


No such 

level 
' C=D 


U 


l-oU 
CZ! 


Jit f nn 

cz> 


101-150 
CZD 


151-200 
CID 


^1-250 


251-300 


300 or 
more. 
CD 


C. Grade 2 


No such 
level 
CD 


♦ 

0 
CZ) 


1-50 
CZD 


51-100 
CID 


101-150 

cz> 


151-200 

cz> 


201-250 
CD 


251-300 
CD 


300 or 
more 
CZ3 


D. Grade 3 


No such 
level 
CD 


0 
C3 


1-50 
CD 


51-100 
C3 


101-150 
CD 


151-200 
CD 


201-250 
CD 


251-300 
CD 


300 or 
more 
CZD 


£. Grade 4 


No such 
level 
CD " 


0 
CD 


1-50 


51-100 
CZ3 


101-150 


151-^ 
CD 


201-260 
CD 


251-300 
CD 


300 or 
more 
CD 


F. Grade 5 


No such 
level 
CD 


0 
CZ> 


1-50 


51-100 


101-150 


151-200 
CD 


201-250 


251-300 


^or 


G. Grade 6 


No such 
level 
CD 


0 


1-50 
CZD 


51-100 
CD 


10M50 

C=3 


151-200 
CD 


201-250 
r ^ 


251-300 
CD 


300 or 
more 
CD 
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2. How has the time specifically scheduled for mathematics instruction changed 
m your school during the past five years? 



Increased 
Substantially 

CZ5 



Increased 
Moderately 



Remained 
Constant 



Decreased 
Moderateiy 



Decreased 
Substantially 



CD 



8. How has the amount of instructional time spent on computational skills in 
grades K-6 changed in your school during the past five years? 



Increased Increased Remained 
Substantially Moderately Constant 



Decreased 
Moderately 

c:3 



Decreased 
Substantially 



-1 



School Name; 



. RS.U. School 
Number Number 



(JLK_13 
CXX3D 

CLXiJ 

ODOD 



CZ3CZ3^0 
CDCXK3D 
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Stamiwd Backj^ound informa^n Form for S-Ysar-Otds 



Does your family get a ne^^jiaper regularly? 
CD Yes C2D • No CZD I don't know. 




Poes your family get any magazines regularly? 



C3 Yes czD No C3 I don't know. 
Are there more than 25 books in youir home? 
CZD Yes cz> No c=3 I don't know. 



Is there an encyclopedia in your home? 

C=> Yes CZD No CD 'TdorTt know. 

How raueh school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school 

father completed.) 

CD Did not complete the 8th grade 

CD Completed the 8th grade, but did not go to high school 

CD Went to high school, but did not graduate from high school 

CD Graduated, from high school 

CD Some education after graduation from high school 

CD I don't know. 

Did your father graduate from a college or university? 



CD Yes CD No CD I don't know. 




How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 

mother completed.) 

CD Did not complete the 8th grade 

c=) Completed the &h grade, but did not go to high school 

C3 Went to high school, but did not graduate from high school 

CiD Graduated ftom hi^ school 

CID Some education after graduation from high school 

cz> I don't know. 

t 

Did your mother graduate from a college or university? 
c=3 Yes c=5 No czD t don't know. 



StMdatd Background Informttion Form for tS-YMr-Olds 



Does your family get a" newspaper regularly? 
CZD Yes CZD No o I don't know. 

Poes your famUy any magazines regularly? 
c=3 Yes c=D No CD I don *t know. 

Are there more than 25 books in your home? 
O Y^, cr> No CZ3 I don't know. 

Is there an encyclopedia in your home? 

CZD Yes No c=>^ I don't know. 

How much school did your father complete? 

<FILL IN THE ONE OVAL which best shows how much school 

father completed.) 

CZD Did not-complete the 8th grade 
CZD. Completed the 8th grade, but did not go to high school 
CD Went to high school, but did not graduate from hi^^ school 
cz> Graduated from hi^ schod 
.CZ) Some education after graduation from high school 

CZD I don't know. 

Did your father graduate from a collie or university? ' 

CZD Yes cz> No cz) I don't know. 

How niuch schod did your mother complete? 

(FILL m THE am OVAL which best shows how much school j 

mother completed.) 

CI3 Did not complete the 8th grade 

CZD Completed the 8th grade, but did not go to high -school 

(ZD Went to h^h school, burdid not graduate from high school 

CZD Graduated from hi|^ school 

CZD Some education after graduation from high school 

* 

CZ3 I don't know.- * 

Did your mother graduate from a coll^ or university? 
cz> Yes CD No CD I don't know. 
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9. Where 4id you live on your ninth birthday? 

C3 la the United Stites (Piease specify the state or territory.) 



Outside the United States (Please specify the country.) 



I don't know. 
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Standard Backgrcxtfid infonnttion Fonn for IT-Yaar-Oldt 



How much school did your father complete? 

(FILL IN THEiME OVAL which best shows how much school your 

father, completed.) 

czD Did not complete the Sth grade 

ciD Completed the Sth grade, but did not go to high school 

CD Went to high school, but did not graduate from high school 

CZD Graduated from high school 

^ome^dueation after graduation from high school 



Idon'fknow. 

Did your father graduate from a college or university? 
CD Yes o No CD I don't know. 

How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 

mother completed.) 

CD Did not complete the Sth grade . 

CID Completed the Sth grade, but did not go to high school 

CZ3 Went to high school, but did not graduate from high school 

CD Graduated from high school 

CD Some education after graduation from high school 



I don't know. 

Did your mother graduate from a college or university? 
CD Yes CD No CD I don't know. 

Wheredid you live on your mntlrbir&day? ~ 
d2 In the United States (Please specify the state or territory.) 

CD Outside of the United States (Please specify the country.) 



CD I ^on't knpw. 
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F. Where did you live on your thirteenth birthd^^y? 

CD In the United Stat^ (Please specify the state or territory.) 



CD Outside of the United States {Please specify the country.) 



CD I don't know. 



Which of the following does your family have at home? (Fiil in one oval 
each ime.) > _ 



Have Db not have 



A. 


Newspaper received regularly 


CD 




B. 


Masrazines received regularly 




t > 


C. 


More than 25 books 






D. 


Encyclopedia 






E. 


Dictionary 


CD 




F. 


Record player 


CD 




G. 


Tape recorder or cassette player 






H. 


Typewriter 


CD 




I. 


Vacuum cleaner 






J. 


Electric dishwasher 






K. 


Two or more cars or trucks that run 


CD 





2. How much time dfd you spend on homework yesterday? 

CD No homework was assigned 

cz> I had homework but didn't do it 

CD Less than one hour 

CD Between ! and 2 hours 

CD More than 2 hours 
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How many different schools have ydu attended since you started the first 
grade? 

cr? 1 to 3 schools 
dD 4 to 6 schools 
cz> 7 to 9 schools 
CZD 10 or more schools 

How long have you lived in the community in which you now live? 
<=D All my life 

CZD 10 or more years but not all my life 
CZD 5 to 9 years 
CZD 2 to 4 years 

CI3 1 year " • 

CZD Less than 1 year 

How much television did.you watch yesterday? 

czz> None cz> 2 hours czd 5 hours 

CZD 1 hour or less cD 3 hours CZ3 6 hours or more 

CZ3 1 hour CZD 4 hours 

Is English the language spoken most often in your home? 

CZ3 Yes CZD No 



Is a language other than English spoken in your home? 

CD Often cz) Sometimes czd Never 

How many brothers_or,dstegs do-youJiav&jaliaare older than you? 
None 1 2 3 4 5 6 or more 

CZ> CZD CZD CZ3 CZ) CZD CZ5 

How many brothers or sisters do you have who are younger than you? 
None 1 2 3 4 5 , 6 or more 

CZD CZ> CZD CZD CZD CZD CZD 

A. What is your racial background? 

cz> American Indiau or Alaskan Native 

CZD Asian or Pacific Islander 

CZD Black 

CZD White 

CZ) Other 
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B. Is your ethnic heritage Hispanic (such as Mexican. Puerto Rican. 

t ufaan. Central or South American or other Spanish Culture or origin}? 

czz) Yes czD No 

How often has each of the following been used in the courses you are taking 
this year? (Fill in one oval on each line.) , 







.Never 


Seldom 


Fairly 
Often 


Free uentiv 


A. 


Listening to the teacher*s lecture 


CZ) 




C3 , 


CD 












* B. 


Participating in student-centered 
discussions 


CID 




CD 


CD 


C. 


Working on a project or in a 
laboratory 


CD 


CD 






u. 


Writing essays, themes, poetry, ^ 
stories 


CZD 




CD 


•* 

i \ 




(>oing on field trips 


CD 


CD 


CD 




h . 


Having individualized instruction 
(small groups or one-to-one with 

a teacher) 


CD 


CD 


< — ) 


CZ) 


G. 


Using teaching machines or computer- 
assisted instruction \ — ^ 


^CD 


CD 


CD 


H 


Watching television lectures 


CD 


CD 


CD 


C^ 


I. 


Studying from textbooks 


CD 


CD 


CD 




J. 


Library or media center assignments 


CD 


CD 


CD 
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Background infomuttion on Matiitmatles Exptritneps for 9>Y«ir*0{cii 



For each of the following questions, fill in one oval in each box. 



A. 


' " ~ , " " '""I 

The metric system uses units like centimeters, liters and kilograms. 
Have you used the metric system of measurement? 

Ves No I don't know. 

(=> CZ> CD 


B. 


How often have you used the metric system in mathematics? 




Often Sometimes Never I don't know. 

CZ3 CZD CD CTD 


G. 


How often have you used a hand calculator? 

Often Sometimes Never I don't know. 

CD C±> CD C3 


D. 


Do you or your family own a hand calculator? 
Yes No I don't know. 

CD CD CD 


E. 


Does your school have hand calculators that you can use in 

mathematics class? 

Yes No I don't know. 


CID dD CZ> 
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Background (nformation on Mathematics Exparienett for tS-Yair-Okis 



For each of the following.questions. fill in one oval in each box. 



D. 



The metric system of measurement uses units like centimeters. liters 
and kUograms. How often have you used the metric system? 



Often Seldom 



Never I don't know, 



B. How often do you use a hand calculator? 



Almost 
Daily 



A few 

times a week 



Less than 
once a week 



Once 
a month 



C. Do you or your family own a hand calculator? 
Yes No I don't know. 



No 



Never 



I don't 
know. 



D^es^your school provide hand calculators for use in mathematics 
Yes No I don't know. 
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Background rnformation on Mathematics Experiences for t7-Y8«r-Old$ 

For each of the following questions, fill in one oval in each box. 



A. The metric system of measurement uses units like centimeters. liters, 
and kilograrjis. How often have you used the metric system of 
measurement? 

Often Seldom Never I don't know. 

CD cr> 



B. How often do you use a hand calculator? 

Almost A few Less than Once I den't 

Daily ' times a week once a week a month Never know. 

CZ) CD CZD 



C. Do you or your fami4y own a hand calculator? 

Yes No I don't know. 

CD CD dD 



D. Does your school provide hand calculators for use in mathematics 
classes? ' 

Yes ■\o I don't know. 



E. Does your school provide hand calculators for use in other classes? 
Yes No I don't know. 
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Which of the following mathematics courses h&ve you studied? Fill in one oval 
oneachline. (If you have not studied a particular course, fiii in the ovai under 
"Not Studied".) 





• 


Studied 1 


Studied % 
avnooi year 


Studied 
less than 
year 


Not 

siuoieo 


I 

don't 
know. 


A. 


General, Business 
Mathematics 


« * 










B. 


Introduction 
(Pre-Algebra) 




CZ5 




CZ3 




c 


Algebra' 










* 


n 


Algebra 




s ) 








E. 


Geonnetry 




f~~i 


r ^ 1 






F. 


Trigonometry 


CZD 


CD 








G. 


Probability & 
Statistics 


CD 


CD 


CD 






H, 


Computer 
Programming 


cz> 






CD 




I. 


Pre-Calculus/ 
Calculus 








CD 


t "' 



r 
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A{»PENDfX D 
NONRESPONSE IN ASSESSMENT SAMPLES 



In addition to sampling variability, es- 
timates of population values computed from 
sample surveys might be subject to random 
error and systematic bias. Systematic bias, or 
nonrandom error, might result from 
estimation procedures, enors inhemit in 
measurement and data collection procedures, 
and nonresponse. This appendix examines 
nonresponse in the 1972—73 and 1977—78 
assessments. Since nonresponse rates at ages 9 
and 13 are relatively smali, the following dis- 
cussion concerns 17-year-olds' re^iponse rates 
only. 

■ Bias due to ncmresponse is present in vir- 
tually every sample suxwy but is fier^uently 
ignored since it is difficult to estimate its aze. 
A variety of factors contribute to non- 
response. Nomvspondents mi^it be either dif- 
ficult to notify or reluctant to partici|mti 
once they are notified; some might be aimnt 
from school during the entire contact period 
with item administrators. However, these non- 
regpondents can be important, since, if they 
respond differently than the people actually 
included in the sample, estimates of percent- 
age based solely on the sample are biased and 
not properly representative 6f the age popula- 
tion being asKsaed. 

To provide some information about the 
siae of the bias due to nonresponse in Nadon- 
al Assessment surveys, the Research Triangle 
Institute, Ralei^, North Carolina, was asked 
to conduct a special study of nonrespond«its 
during the 1972-73 assessment of science 
and mathematics. The study was conducted 
on the age population of eligible 17-year-olds 
who, at the time of the assessment, were list- 
ed as ofjroUed in school. Some of these stu- 
dents, in fact, were no longer attending school 



at th6 time of the ass^sment. Eligibles had to 
be English-speaking, had to be i^ysically and 
eraotionaily able to respond to exercises as 
administ^ed, and could not reside in an in- 
stituticm. 

The results of the nonresponse, study 
CKalsbeek et al., 1975; Bogers et al., 1977) 
indicate that 17-year-oIds listed as enrolled in 
schools but not appearing at the designated 
time of ass^ment can be divided into two 
different grou{». The first group of non- 
respondents, which comprises about 80% of 
the total ncmrespojident group, did not 
appear for the assessment b^»^ of con- 
flicting school acf^ti^ or iUo^.- The per- 
formance of this group was not very different 
ftom the p^omiance of students mmtcd 
during the r^ularly scheduled sessioaf. The 
second group of nomespondents, n^ch com- 
prises ahmt 20% of the nonxespondents, do 
not appear to be aWOable in the sdiools at 
any time. They attehd infrequently if they 
attend at all (for pracfecad^lHi^oses, they have 
dropped out of sdiool), or they have moved 
out of the school att^tdaiice am. In either 
case, tl^ students sfaoiild probably not have 
been listed in the in-sdiool popuUtion .of eH- 
gibles. This group. In contiast to the group of 
nonrespondents who were, in fact, attending 
school, performed more pooriy on assesnnent 
qu^Ucms tlun students assessed during the 
sd^uled sessions. 

The weights used by National Assessment 
to estimate the perc^ti^ of acc^stable re- 
sponse are «^jutted for nonresponse. The ad- 
jus^nt assume that the nonraapon^ts 
would perform, on the average, ih a manner 
similar to those who did respond. However, 
the nonretpcmse study showed that the 
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second group of nonrespondents, those en- 
rolled in but not actually attending school, 
typii*al|y performed at a lower level than 
either those who did respond or the first 
group of nonrespondents. If the second group 
is* included in the population of eligibles, the 
nonresponse- adjustment procedure used by 
National Assessment would result in over- 
estimates of the true percentages of accept- 
able responses. 

Because the second group of students is ef- 
fectively no longer attending school, it does 
not seem appropriate to include them in es- 
timates for IT year-oids in school Thus, these 
students are not considered part of Uie popu* 
lation of eligibles and are excluded from the 
computations of percentage of the sample 
covered for 17-year-oids, as shown in Table 
D-1. 



Including the second group of students and 
then reducing bias tJue to their nonresponse 
would require the location and testing of 
some of these individuals. The difficulty and 
costs associated with supplementary data 
collection of the nonrespondents not actually 
attending school are so great that this has not 
,been a feasible alternative in recent years. 



Table D-1 shows the average sample cover- 
age per package (booklet) of exercises ad- 
ministered in 1972-^73 and 1977-78. The 
rate of coverage is based on an estimated total 
eligible age population of students who are 
available in school — for 17-year-olds, those 
enrolled minus the 20% estimated to be en- 
rolled but unavailable in school. 



TABLE D t. Numbftf of Students Assessed and Percmit of Sample Covered, by Age and Ass^ment Year 



Year 



1972 73 



1977 78 



Age 



9 
13 
17 

9 
13 
17 



ISiumbar of 


Total Number 


Average Number 


Average Sample 


Packages 


of Students 


Assessed Per 


CoMrage in 




Assessed 


Package 


Percent 


7 


18.638 


2,663 


- 91.0 


9 


23,507 


2,612 


84.6 


11 


25.865 


2,351 


77.7 


7 


17.190 


2,456 


87.2 


11 


26.661 


2,424 


85.2 


12 


26,757 


2.230 


73.2 



Note: .Computations for U year aids do not intlude those enrotlea in school but not actuaf/v, attend- 
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^. APPENDIX E 

NATIONAL ASSESSMENT ESTIMATION OF STANDARD ERRORS 



Several measures of . achievement that 
National Assessment uses in its reports are 
described in Chapter 7. The sample design 
described in Chapter 4 is a complex, deeply 
stratified, multistage probability sample de- 
sign. A reasonably good approximation of 
standard error estimates ot these achievement 
measures can b«^ obtained by applying the 
jack knife procedure to first-stage sapling 
units within strata, using the method of suc- 
cessive differences and accumulating across 
strata. 

'• " ■ ' . •• 

In ttiis section, the measures of achieve- 
ment are first defined in algebraic form, fol- 
lowed by a description of the jackknife 
method used by National Assessment to es- 
timate their standard errors. 



Metres of Adifevttnent 

Based on the sample design, a weight is 
assigned to every individual who responds to 
an exercise administered in an assessment. 
The ' weight is the reciprocal of the probability 
of selecting a partic ular individual to take a 
particular exercise with adjustment for non- 
response. Since the probabilities of selection 
are based on an estimated number of people - 
in the target age population, the wei^t for an 
individual estimates the number of similar 
people that that individual represents in the 
age populaition. As explained in Appendix F 
the weights were adjusted to refiec,t in- 
formation from previous assessments on 
population distril^tions. 

A sum of the weights for ail individuals at 
an age level responding to an ex^cise is an 
estimate of the total number of people in that 



age population. A siim of weights for all indi- 
viduals at an age respqnding correctly to an 
exercise is an estimate of , the number of 
people who would be aWe to respond ccarect- 
ly in the age population if the entire popula- 
tion were ass^sed. These concepts also apply 
to any reporting group (e.g., defined by re- 
gion; sex, etc.) and cat^ory of response (e.g., 
corr^t, incorrect aiid "I don't know"). 



ihk 



sum of wei^ts for respon- 
dents to exercise e who are in 
reporting subgroup i and who 
are in the &th PSU of the ftth 
sampling stratum, and 

sum of wei^its for r^pon- 
dents to exeitnse e who are in 
subgroup i, who are in the feth 
PSU of stratum h and who se- 
lected response category j 
(e.g., correct foil) for the exer- 
ci^. 



cf4- 

Then, summing K over the n^, sample PSUs 
in the stratum h, and aimming over the H 

N^ampling strata, - £ s'* W^.., 

estimates the number of eiigibles in the popu- 
lation who are in sul^roup L 



Similarly. Cf^ - ? 2^ cf/. 



esti- 



mates the number of eUgiMes in tfae popula- 
tion who are in subgroup j and who would se- 
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iect response cat^ory J for exercise e. 

Ai? estimate of the proportion of the eiigi- 
bles in the age population in group i who / 
would select response category / on exercise e j 

is: ■ ^ 

(1) - / wf„. 

In the special case where the proportion of 
all ^e eligibies who would select response 
category ; on exercise e is estimated, the in- 
dex A (for ALL) will be used in place of i as 
follows: 

■ (2) - ''>^A**- 

In National Assessment reports, the pro- 
portion in (1) multiplied by 100 is called the • 
group percentage, and the proportion in (2) 
multiplied by 100 is called the national per- 
centage. The difference between the pro- 
portion in subgroup 2 who would select 
category / on exercise e and the proportion in 
the nation is denoted by: 

(3) - ff^ - l^l 

National Assessment also reports the arith- 
metic mean of the percentage of correct re- 
sponses ovet sets of exercises corresponding 
to the measures in (1), (2) and (3). These 
means are taken over the set of all exercises or 
a subset of exercise classified by a reporting 
topic or content objective. The mean percent- 
^age of correct responses taken over m 
excises in some set of exercises cori^^ond- 
ing to measures (1), (2) and (3) are, respec- 
tively: 



(4) P, 



(5) P 



me* 



^ 2 C^^^ / ^Vl^ and 



(6, .-.p.-p^-p^. 

Note that the response category subscript ; 
has been Ktppressed sin<» the means are 



understood to be taken over the conect le- 
sponse cat^ory for each exercise. 

Each of these six achievement measures, Is 
computed and rou;tineIy izsed in reports 
describing achievement data for any ai^ess- 
ment. The dmple difference in these measures 
between two s^sessments of the same ezehuse 
(or sets of exercises) provides six measured of 
change in achievement that are routinely u^ed 
in National Assessment's change reports. The 
next section describes how standard errors are 
estimated for the 12 statistics routinely used 
in NAEP reports. 



CtHnputatioii of Standard Errors 

In order to obtaih an approximate measure 
of the sampling variability in the statistics (1) 
throu^ (6), a jackknife replication procedure 
for estimating the sampling variance of non- 
linear statistics from complex, multistage 
sample was tailored to National Ass^ment*s 
sample design. References (4), (5) and (7) 
provide information about the jsuskknife tech- 
nique, while reference (3) describes how the 
-procedure is used in estimating standard 
errors for National Assessment's sample de- 
signs. 

To demonstrate the computational aspects 
of this technique, consider estimating the 
variance of. the statistic in (I) — the pro- 
portion of age-eligibles in sui^oup i who 
would select response ^tegory j on exercise e. 

This statistic is based on the data from all 
the PSUs in the i# strata. Let P^Lm, ^ 
defined as a repUration estimate of p^and 
constructed from all the PSUs, excluding the 
data from PSU k in stratum h. These 
replication estimates axe computed as if the 
excluded PSU had not responded and a 
rieasonable nonre^onse adjustment is used to 
replace the data in PSU hk in estimating pV, 
Several choices for replacing the ciata in PSU 
hk are available. In order to obtain a 
convenient and computationally efficient 
algorithm for approximating standard enors. 
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t National Assescment repl»^ C^^ and 
from the ftfcth PSU with corresponding sums 
froa another paired PSU in the sai&e stratum. 
The repUsJate ^timate is then computed. Hje 
replicate estimates to be used in the 
caictUations are determined by arranging all 
the VSVs in each stratum into successive pairs. 
That is, PSU 1 is^ paired , with PSU 2, PSU 2 
with PSU 3, 3 with 4. .. . (n.~l) with n^, and 
PSU with PSU 1. ^ . 

• 

The contribution to the variance of pV fay 
each pair of PSUs is the change in &e value of 
the statistic incurred by replacing the data 
from each PSU in the pair with the data from 
the other PSU in the pair and recomputing pf^ 
in the bsual way. This produces two repUcale 
estimates. . Squaring the di^erence between 
. these replicate estimates and then dividing by 
eight measures the contributibn of this pair of 
PSUs to the total variance. The sum of these 
contributions over all n. successive i^ii« in 
the stratum is the contrmution by stratum h 
to the total variance. The square root of the 
sum of the W stratum contributions is the 
estinoiate of the standard error of pf/ 

Algebraically, the two replicate estimates 
for the pair k, k-f-l (where ... n, and - 
are: " 



and 



The contribution to the total variance from 
stratum h is: 

(9) var (Pfl) - i ll'fML^ - V 
^ 8 k \ i~fik U-^(k+liJ 

And, finally, an estimate of the standard 
error of pf is: • 



(10) SE (P^j) -i (Z var ■ 

Multiplying pf by 100 yields the percent- 
age of rei^onse to category /. Multiplying SE 
(pf) by 100 yields the correqKmding esti- 
mated standard error of the percental. 

In general, the jackknifed standard errors 
of the proportion estimates will be larger thm 
the simple rpdom sampling formula/po/nj^ 
where p^pf, ^-i-p and n is the number of 
sampled re^ndents in subgroup i viho toojs 
the exercise. The larger the size of (pV) 
reflects mainly the loss of precision due to 
cluster-sampling of schools and students. 

The standard errors for the achievement 
meaaires (2) throu^ (6) are cdmputed' 
throu^ a series of steps analogous to those 
foUowed in •ompu^'sr fpf;. The most 
complicated- step in eompueiog standard 
errors occurs in formii^ the paired rejrticate 
estimates analogous to (7) and (8) for au:h 
sua^ssive pair of PSUs. Once this bookkeep- 
ing chore is done, the computaticms (9) and 
(10) foHow in a sfcrai^tforward manner. 

The standard errors for the differences be- 
tween two assessments for any of the achieve- 
. ment measures (1) throu|^ (6) are computed 
as the square root of Oie sum of the sqmu«d 
standard "enors from each of the a^jjar&te 
ass^ttments. 

The size of the standard errors depends 
largely not only on the mmiber of PSUs and 
school* inclutted in the sample (Table ^, 
Chapter 4), but aUo on the number of re^ 
spondents in each of the resorting groups. 
Table E-1 shows the average number of stu- 
dents respcmdhig to an exercise pulage for 
each of the reporting groups for ^ch «ge and 
for eac^ of the two mathematics asMisments. 

The size of the standard enorv of the 
means of tlie achievement n^suies for sete of 
exerdses is also mfiuenced by the number of 
ex^dses in the exerdse set and the number' 
of packages over whidi the items in the set 
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arp spread/Table E-2 shows the number of 
exercises and package included in the mean 
achievement measures used to measure 
changes between mathematics assessments for 
> each of the cognitive-process levels. 
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TABLE E<t. Avtrsge AiunUitr of RMpondentt in. Reporting GitHipi 
Taking an Item Socket, by Age and Assessment Year 



A»9 

1972-73 1377-78 



Age 13 

1972-^73 1977-78 



Age 17 

1972-73 1977-78 



Nation 

Region 
Northeast 
Southeast 
Centrsi 
West 

Sex 
Mate 
Female 

White 
Black 

Parental education 
Not graduated high school 
Graduated high sdvsoi 
Post high school 

Typf of community ^ 
Extreme rural 
Disadvantaged urtjan 
Advantaged urban 

Size of community 
SigdtMK 

Fringes around big cities 
Medium dties 
Smaii place* 

Grade 

3. 7. 10 

4. 8.11 
12 



2.663 



656 
663 
672 
665 



1.328 
1,335 



1.997 
466 



2i1 
564 
787 



265 
266 
267 



619 
515 
372 

1.157 



646 

1.946 



2.456 



552 
591 
764 
549 



1,234 
1,221 



1.843 
433 



204 
579 
772 



240 
245 
247 



640 
524 
170 
1,122 



634 
1.755 



2.612 



651 
667 
649 
845 



1.294 
1.318 



1.977 
436 



417 
792 
994 



263 
264 

mi 



583 
531 
365 
1.133 



693 

1,809 



2.424 



547 
546 
770 
561 



1.210 
1.214 



1.862 
396 



302 
797 
942 



242 
241 
243 



685 
519 
170 
1.049 



652 
1,704 



2.3S1 



573 
596 
596 
586 



1.126 
1,225 



1,852 
358 



455 

720 
1,028 



230 
239 
234 



439 
493 
326 
1.094 



SOS 
1.6^ 
304 



2,230 



516 
509 
731 
474 



1.085 
1,145 



1.796 
306 



334 
763 
1.020 



221 
214 
227 



612 
483 
126 
1,008. 



1.675 
2^ 
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TABLE E 2. Numbtr of Emrctsts In Varioia CognitiM-Prootss Lwtfs Uitd To Mnsttrt Chmot BitwMn t972-73 
1977-78 and Number of Pudeagis in Which Tho» Extrcisas Apfiaartd in E«sh As^tetment. by Aga 



AM axf rcises 
Typa of mathamatlcs 
Skills 

Undarstanding 
Applications 



of of 
Exarcisai Padc^M 

1972-73 1977-78 



55 


7 


6 


17 


4 


4 


21 


7 . 


6 


8 


5 


5 


9 


4 


4 



AgstS 

Numbfr Numbtr Numbar 

of of of 

Ejurdsw PidugM Ex«rds«s 

1972-,73 !977-m 



Agt17 

Nundjwr 
of 

1972-73 1977-7 



77 


9 


9 


102 


11 


11 . 


16 


i 

4 


4 


18 - 


8 


8 


37 


9 


9 


46 


11 


11 


12 


8 


8 


13 


9 


9 


12 


7 


7 


25 


10 


10 ' 
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' ' APPENDIX F 



ADJUSTMENT OF RESPONDENT WEIGHTS 
AND ESTIMATED POPULATION PROPORTIONS OF REPORTING 
GROUPS BASED ON NATIONAL ASSESaVTENT SAMPLES 



A« noted in Appendix D, a w^i^t is 
assigned to ever^ individual who responds to 
an exercise administered in an assessment. Be. 
cause the weight is the reciprocal of the prob- 
ability of selection of the .individual with ad- 
justment for nonresponse, and since the prob- 
abilities of selection are bJised on .the 
estimated nuinber of people in the target age 
populations, the weight for an individual es- 
timates the number of similar people. that that 
individual represents in the age pofsalation. 

The estimated pojpulation proportions for 
reporting gro\ips shown in this appendix are 
based on wei^ts derived from the sampUng 
process used in the two assessments of 9«, 13- 
and in-school IT-year-olds. These proportions 
vary from year to year because of midom 
sampling variabihty or systematic differences 
in sampling pro<»dures, A better estimate of 
population proportions, for any single year 
can be obtained by smoothing* the pro- 
portions over several assessment years. 
Smoothing does not make thf Ktimated pro- 
portions identical but does reduce variability. 
The estimated population proportiom i^own 
in this appendix and used in estimating per- 
formance were obtained after smoothing pro- 
portions from the first nine years of assess- 
ment. The procedures used to obtain the 
smoothed proportions are detailed below. 



^ Tlw word "vnootiiing" it uicd bnm in tb* mom of 
drawing * "irnooth" curv* to fit a -Mqucnct of 
aumtwn. Proportioiu f<^ Mch rtp<»tiaf group covtr- 
iaf •^^t yetn wn* «mo(^«d bj^ Ih* robui^ncisUai 
peoewiunft {Tukmy, 1977, Chipter 7). 



The purpose of smoothing estimated popu< 
lation proportions is to reduce «ft?n pling 
fluctuati^is that can affect ^timates of the 
change over time in the pen;enta^ of accept- 
able responses to an exercise. Fcur example^^ 
the percentage of acceptable r^pons^ for an 
age group is a function of the relative pro- 
portions of high -performing and low- 
performii^ groups. If the relative p»>p(»rtions 
of these groups are^very different in different 
assessments b^ause of sampling variability, 
then a portion of the change in perc^trge of 
acceptable responses an - age group is 
directly attributable jto the yearly sampling 
'difference in the relative proportions of high- 
and low-achieving groups. Smoothing es< 
timates of population proportions reduces a 
laiiie portion of the sampling variability while 
preserving, as far as p^ble, actual traidi 
occurring in the age population. 

Besides report performance estimates: 
for an age groitp as a whole* NaMcmal Assms-' 
ment also repcvrts pei^ormance for various 
subpopulations, su^ as whites or blacks. Be- 
' cause \^uiability of iubgroup« within thMe 
subpopulati<His (si^ ai white malet and 
female within the itiiite subpopulatimi) in- 
flueiH^ perfonoance estimates for the 
subpopulatioos, it is desiriabte that fluctua- 
tions of laroportiona of all nibgiouiw of eftch 
subpopulation be reduced as much as 
poi£ble. 

The mo«t direct way to acccmipliih this is 
fint to dataify people into mutually esdu^ve 
multiway cellf on the bans of their memb^. 
fhip in categcufiea of vaxiom important 
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vambles and then to smooth the proportions 
within each of the resulting muf ti way c^lls 
across years. Unfortunately, this procedure 
tends to produce a large number of cells with 
few people and consequently, quite unstable 
estimates of "smoothed proportions. To^ cir- 
cumvent this difficulty, tl^e smoothing pro- 
cedure, which is basically a wei^ting-class ad- 
justment applied independently to each age, is 
designed to control, to varying degrees, fluc- 
tuations in certain key subgroups while avoid- 
ing, as much as possible, instabilities due to . 
small ceils. The procedure involves two dis- 
tinct levels of partitioning of the population. . 

The first step in the weight adjustment pro- 
cess involves partitioning the population of 
age class eligibles into 672 six-way cells oh the 
basis of membership in categories of six key 
demographic v&riabl^: race, grade, sex, size 
and type of community, region and parental 
education. The categories of th^ variables • 
that were used in the partitioning are present- 
ed in Table F-1. Then, for each pf the three 
ages and each of the nine years, the pro- - 
portion, of the population in each of the six- 
way cells was estimated. For a given age and 
year, the proportion of the population in a 
particular six -way cell was computed as the 
simi of weights of all respondents assessed in 
the given year who were of the specified age 
and who belonged in the six-way cell divided 
by the total of the wei^t of all respondents 
of the given age assessed in that year. 

The second step in the weight adjustment 
process is the aggregation of the six -way cells 
into 17 larger cells, called major cells. These 
major ceils, which comprised the first leveUjC^^^^ 
partitioning for weight smoothing, were t|je 
same for all three ages and all nine years and 
are detailed in Table The purp<^ of the 
aggregation to m^or cells is to obtain sub- 
groups of the populations that have ^aistoric^- 
ly exhibited substantial differences in per- 
formance and that define groupinj^ for which 
the most control of the weight was desired. In 
order to ensure stability in tl^ weight s^just- 
ment process, each major cell was required to 
contain an adequate number of respondents 
for each age and year« 



TABLE F-1. Cattgortis of tht V«riibtt$ 
Ustd in the Construction of Six-Way CiUs 

VarN^tts Cittgoriei 

Race White, black, other 

Grade Less than modal grade, roodai grade 

or greater 

> 

Sex Male, teitiaie 

Su0 and Extreme rural, disadvant^jed urban, 

type of advantaged urban, main cities* 

• community urban fringes, medium cities, 
small places 



Region 



Northeast, Southeast, 
Central, West 



Parental Graduated high school and less, 

education post high school 



To exert control on the six-way c^Us within 
e§ch of the m^jor cells, a version of the 
Smear-and-Sweep promiure developed in the 
Nationai Halothane Study (Bunker et al., eds., 
1969) was used« Nationd Assessment's use of 
this procedure involved ordering the six-way 
cells within a ms^or ' cell by predicted 'per- 
formance and then a^iegating adjacent six- 
way cells into superc^lls. Each supercell was 
required to represent no less than 1% of the 
total populaticm in each y^. The ordering of 
six-w£¥ eeUs and a^^Wting into superceUs 
was done independently for each age. There 
were between 1 and 80 six-way cells in a 
supercell and between I and 16 supen^Us in a 
mai&t ceil. There were a total of 66, 74 and 
65 supercells created for ages 9, 13 and 17* 
mpectiTOly. 

The partitdoning of the mi^or cells into 
supercells allowed some dampening of the 
fluctuaticms of proportions of the six-way 
cells, and hence, of all one-way mazgirn. At 
the same time, instabilities of adjustment due 
to small cell size was reduced by using super- 
cells instead of the dx-way cells. AdditicmaUy, 
by aggregating six-way cells with dmilar per- 
f omiance, bias in perf ormande estimates using 
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TABL£ f -2. M«|CK Cails UMd m tiM Adjitttmtnt of Poputetkm PrafxsrtiOni 





RMt 


Re(^* 


Size and 


Grtdi** 


Sex 








Type of 






* 






CMnnKinityt 






1 


White 


S£ 


OU • ' 


AH 


AH 


2 


White 


SE 


AU 


AU 


AU 


3 


White 


SE 




GEMG 


Mate 


4 


White 


SE 


not OUAU 


GEMG 


, Female 


5 V 


White 


SE 


not DU^U 


LTMG ^ 


Aii 


6 


White 


notSE 


OU 


AU 


All 


7 


White 


notSE 


AU 


A!! 


AH 


S 


White 


rxjt SE 


Not DU^U 


GEMG 


Male 


9 


White 


not SE 


not DU^^U 


GEMG 


Femaie 


.10 


White 


notSE 


not OU>\U 


LTMG 


Mate 


11 


White 


iKJtSE 


not OU^U 


LTMG 


Female 


12 


Bfack 




OU 


AH 


Alt 


13 


BSack 


SE 


not DU 


M\ 


Ait 


14 


BSack 


trot SE 


OU 


AW 


Ati 


15 


Black 


SE 


not DU 


AH 


Ait 


16 


Other 


Ail 


OU 


AH 


Ail 


17 


Other 


' AH 


not OU 


Alf 


Aii 



tDU^fsadvantag^d urban; A U^adv^t^gtfd urbw, 

• *C£AIG«madS9/ gra(h or gr^ur; L TA^GWcss thsn mad^ grad^. 



s^moothed proportions was reduced* 

The next step in the smoothing procedure 
is the actual smoothing of the supercei! pro- 
portions. For ^ch 8ge, there was a set of nine 
proportion Tor each supatrell, representing 
the proportion of people in the supercell for 
each of the nine assessment y^rs. 

Each such set of proportions was then 
smoothed by roiHlSt/r1Misitl^lt methods to give 
a se q u ence of adjusted populadon pro* 
portions that tended to pi\aserve actual time 
trends in prc^ortions while redudng the 
sampling variability of these estimatef^ over 
time. The adjusted proportions were ayn- 
straimd by requiring that the sum of adjusted 
proportions across ail supercells for eiu:h age 
and year total one. 

The final step in the smoothing procedure 
is to adjust the respondents* weii^ts to be 
cdin^stent with the smoothed superc^ 
portion*. Since e^di retp«mdent takes txdy 



one package, the weight adjustments were 
done independently for each package. For a 
given age, year and package, populaticm |ffo* 
pc^tions losing tttB original wei^its ob- 
tained fox fach of the superoelli api^priate 
for the age. Then the wel^ts of ail re- 
spondents of a 0xea mpet^ wa« adjusted 
by multiplying by a factor that is the ratio of 
the smoothed supexcell i^oportion divided by 
, the^fnoportion using tl» original waists. This 
produced the ^tlfusted wei^ts that wezte used 
in aUanalj^. 

The population proportions pzwM»ted is 
Table F-3 were c<^puted using these adjusted 
weights.. All performince estimates were also 
computed ui'^ the adjusted weights. As ex- 
plained in Appttadix E, the percental of 
correct response is ^timated by (fividing the 
sum of w«l^ts for stuctents resposdiag 
correctly to an exM^ise by tlw sum of w«l0tts 
tor all students exposed to the tzezdsif. For 
example, in 1977, 11 i»ckages of ex^dsee 
wi^ admiiiisti»«d to IS-year-olds. Smoothed 
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population proportion estimates were based 
on 26,661 13-year-olds, but each exercise- 
s|>ecific jierformance estiniate was based on 
the approximately 2,424 13-yenr-o!ds who 
took a particular package. Consequently, re- 
spondent wei^ts were adjusted to dampen 
b«th between-year variability and package-to- 
package variability within an asse^ment year* 



The result of the smoothing and weight- 
a<^ustment prcK^ess is that both adjusted per^ 
formance esti'mates and changes in thc^ es- 
timates appear to be somewhat less 
susceptible to sampling variability, both 
across and witliin years. 



TABLE F 3. Ettinu^ Foputotton Proportiofis of Nationiri Asselsmtnt Rtporting Groups 
for A«n 9, 13 and 17 in 1S72-73 and 1977-78 



r 



Reporting Groups 

Sex 
Male 
Female 

Race 
White 
Black 
Other 

Region 
Northeast 
Southeast 
Central 
West 

Parental education 
Not graduated high school 
Graduated high school 
Post high school 
Unknown 

Type of community 
Extrenw rurai 
Disadvanta^d urbah 
Advant^ed urban 
Other 

Si2e of community 
Big cities 

Fringes around big cities 
Medium cities 
Small places 

Grade in school 
<3,<7,<10 

3, 7, to 

4. a, t! 

<4,<S, 12 
Other 



• 


1972-73 






1977-78 




Agt9 




A«8 17 


AgcS 


AQt13 


Ab«17 


.500 


.498 


.484 


.502 


.502 


.484 


.600 


.502 


.516 


.498 


.498 


.516 


.808 


.816 


.849 


.799 


.811 


.832 


.133 


.127 


.109 


.133 


.125 


.113 


.059 


.057 


.042 


.088 


.064 


.055 


.257 


.241 


.259 


.239 


.236 


.240 


.223 


.220 


.199 


.234 


.224 


.205 


.280 


.300 


.311 


.289 


.292 


.314 


.240 


.239 


.231 


.238 


.248 


.241 


.096 


.146 


.175 


.089 


.132 


.148 


.222 


.302 


.320 


.245 


.321 


.326 


.320 


.416 


.458 


.328 


.420 


.480 


.362 


.136 


.047 


.338 


,127 


. .046 


.VJOO 




.Uo4 


.0S4 


.098 


.085 


.074 


.074 


.090 


.075 


,067 


.085 


.122 


,114 


.114 


.106 


.109 


.111 


.713 


.720 


,712 


.735 


' ,726 


.719 


.211 


.194 


.181 


.190 


.200 


.173 


.227 


.233 


.250 


.213 


.213 


.235 


.141 


.134 


.141 


.138 


.135 


.142 


.421 


.439 


.428 


.459 


.452 


.450 


.010 


,027 


.017 


.006 


.021 


.015 


.223 


.246 


.126 


.233 


.254 


.133 


.755 


.716 


.729 


.751 


.716 


.752 


.006 


.010 


.127 


.008 


.0(» 


.100 


.006 


.001 


.001 


.004 


.CK)1 


.000 
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GLOSSARY OF NATIONAL ASSESSMENT TERMS 



Acceptable response. Any response to an 
exercis*" that demonstrates achievement of 
the objective measured by that exercise. 

Administration time. The total time allowed 
on the paced audio tape for an exercise. 
(Includes the time allowed for the stimulus 
and the response.) 

Administration timetable. Time periods 
during the school year when the various age 
groups are assessed. For the matliematics 
assessment the timetable was as follows: 



Age Pint Mathematics 
Aiia^uncnt 

9 January to Febnury 

1973 

13 October to December 

1972 

,17 March to ^fay 1973 



Second Mathematics 
Assesin)«nt 

January to February 

1S78 

October to Oc<»{nb«r 

1977 

March to May 197S 



In the first mathematics assessment, young 
adults were assessed from October 1972 to 
May 1973. 

Age group or age level Three age groups were 
sampled in the 1977—78 assessment: 
9-year.olds, 13-year-olds and 17-year-olds. 

In 1972-73 young adults aged 26—35 were 
also sampled. For the 1977-78 assessment, 
the three age groups were defined as 
follows: 



Age 9 
196S 



bom during the calendar year 



Age 13 — bom during the calendar year 
1964 

Age 17 - borh from Octobb« 1, 1960, 
through September 30, 1861 



Assesment, The documentation of the pro- 
gress in knowledge, skills and attitudes of 
American youth. Measuies axe takm at 
periodic intervals for each learning ai-ea, 
with the goal of determining trends and re- 
porting the findings to the public and to 
the education community. 

Assessment adminiatmtor. Individual employ- 
ed to administer the assessment in 
participating schools. 

Attitudes. The mathematics assessment sur- 
veyed attitudes toward mathematics in 
school, mathematics in society, math- 
ematics as a discipline and mathematics in 
relationship to oneself. 

Background qu€8tion&. Quei. ■^•'t specific to 
mathematics wez€ include-" ^ the item 
booklets and dealt with rer :ents' edu- 
cation and experience in iidtl^matics. 
Standai^ background questions asked in 
every learning area are found on the back 
pages of the it^ booklets and include such 
thin^ as level of |»rental education and 
reading materials in the home. Samples of 
these questions are foimd in Appendix C. 

Boofctef. Items Cexearcises) are pn^nt»i to re- 
spondents in booklets. Booklets are de- 
signed to be scared by optical scanning 
machines. Each booklet contains (1) in- 
structions on answering items and sample 
items, (2) assessment items and (3) back- 
ground questions. Each booklet contains 
aiH>roximately 30—35 minutes of assess- 
ment items and 10—15 minutes of in- 
troductbry mat^ial and background ques- 
tions. A booklet typically includes exer- 
cises of varying difficulty from different 
objectives of the learning area bdng 
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Cuiculator booklet. An entire booklet at each 
age level was deigned to be answered imng 
a four-function electronic hand-held cal- 
culator. Calculators were issued to students 
taking these booklets. 

Calculator item. An item in a calculator bv.ok- 
let designed to be answered using a hand- 
* held calcuiator, 

Category iscaringi, A classification of a re- 
sponse to an open-ended exercise. See 
scoring guide. 

Category within a inxriable. A subclassification 
within a variable. For example, male and 
female are categorie*s of the variable, sex. 
Also see reporting groups. 

Cognitive-process level. Diflerent types of 
mental processes required for mathematics 
problems. These levels form t^ns dimension 
of the two-dimensional matrix used to 
structure the mathematics assessment. The 
four cc^nitive-process levels defmed for the 
mathematics asf^essment are: knowledge, 
skills, understanding and application. 

Content category or area. Classification of the 
content included in the mathematics assess- 
ment. Content forms one dimension of the 
two«dimensional matrix used to sUiicture 
the mathematics assessment. The five 
content categories defined w«re: numbers 
and numeration; variablts and relation- 
ships; geometry (shape, siae and position); 
meoiiurement; wid other topics (including 
graphs and tables, proi?ability and statistics, 
and logic), 

Exerci^. A task desiipied to measure an ob- 
jective. Because NAEP does not administer 
*'tests/' but instep describes education 
ac^evement over time, thr term **exercise'* 
is often used instead of the t^m 'item** or 
**te*t item.'' The term* '*it«m'' and Vexer- 
cise'' are used synonomously in ibis report. 

Exercii^ booklet. See booklet 



Exercise part See item part. 

Exercise pooL The entire set of exercises pre- 
pared for a learning area. This set includes 
recycled exercises, exercises developed for 
previous assessments but not used due to 
exercise booklet or budgetary constraints, 
and newly developed exercises. 

Field tritds (tryouts). A pretest of exercises to 
obtain information regarding c!arity, dif- 
ficulty levels, timing, feasibility and special 
administration problems needed for re- 
vision and selection of exercises to be used 
in the assessment. 

Grade. Results are reported for 9-yearH>Ids in 
the 3rd and 4th g^^, 13-year-olds in the 
7Ui and 8th grades, and 17-year-oIds in the 
10th, lith and 12th grades. 

Grou^ administratic^. Booklets were adminis- 
tered to groups of between 10 and 25 stu- 
dents in 1977—78, In 1972^73, group size 
was 12 respondents. 

Hand scoring (scoring), Thp coding of re- 
sponses in a format compatible %vith the 
optical scanning equipment being lised. 
Multipie-cho!(^ exercise can be directly 
machine scored; however, responses to 
open-ended, exercises must be coded in 
scoring ovals so that they can be machine 
scored. See scoring guide. 

ID number. An identificaition number. Usual- 
ly refers to the unique number assigned to 
each respondent This number is as^ipied 
to pr^rve the anonymity of each re- 
spondent* NAEP does not ktrep records of 
the nam^ of any individuals. 

Item. See exercise. 

Item bfK^klet See booklet 

Item part. Each p^rt of an item asks a sepa- 
rate ques^on. Farts may ail pertain to one 
stimulus, such as a g^h or a table^ or may 
instead concern the same topic. In the 
matl^matics ass^sm^t^ each item part 



was analyzed as a separate unit of data, so 
counts for various content categories and 
cofjnitive-process levels are done by item 
part Same as exercise part 

Learning area. One of the 10 areas assessed by 
National Assessment: art, career and 
occupational development, citizenship, 
literature, mathematics, music, reading, 
science, social studio and writing. 

Level of parental education. Three levels of 
parental educaMon are reported. These 
levels are desc*.:^- in Appendix B. 

Mathematics assessment The content domain 
for the second assessment of mathei^atics 
draws primarily from ^ the current cur 
riculum of elementary and secondary 
schools, although some projection of future 
mathematics emphases is included {for 
example, greater use of metric measure, 
earlier introduction of decimals and cal- 
culator computation). Mathematics up to, 
but not including, Calculus is included in 
the assessment. 

Medal grade. The grade in which most of whe 
students at an age level are enroDed. For 
9-year*oids, the modal grade is the 4th 
grade; for 13 year-olds, the 8th grade; and 
for 17-year-oids, the 11th grade. 

Objective. A desirable education goal agreed 
upon by scholars in the field, educators and 
cor;' ?med la-, persons, and established 
through the c ; ansus approach. 

Objectives redevelopment. After the initial 
ass^ment of a learning area, one of the 
first steps in preparing for reassessment is a 
review of the learning-area objectives. This 
is carried out by scholars in the field, edu- 
.cators and concerned lay persons. These re- 
views may result in revision, modification 
or total rewriting of the leaming-area ob- 
jectives to reflect current eurricular goals 
and emphases; they m^y also r^ult in the 
endorsement of t}^ oajectives tram the 
previous ass^ment as totally adequate for 
the next assessment. 



Open-ended exercise. A non-multiple^oice 
exercise that require some type of written 
or oral response. 

Paced audio tape, A tape recording that 
accompani^ each booklet to assiire imi- 
fbrmity in administration. Instructions and 
exercises are read fay the announcer onto 
the tape so that reading difficulties will not 
interfere with an individual's ability to re- 
spond. Response time is included on the 
tape. 

Principalis questionnaire. A data collection 
form given to school principals. The prin- 
cipals respond to qu^Uons concerning en- 
rollments, si2e of the community, occupa- 
tional composition of the community, etc. 
See also supplement^uy principalis ques- 
tionnaire. Sample* of these qu^tionnaires 
are found in Appendix C. 

PSU,, A primary sampling unit. This is the 
basic geographic sampling unit for National 
Assessment. A PSU is either a single county 
or a ^t of contiguous coimties. 

Public u6e data tapes. Computer tap^ con- 
taining r^pondent-leve! data for exercises. 
Th^ t^>es are available for use by ex- 
ternal researchers wishing to do secondary 
analyse of National Assessment <&ta. 

Racial/ethnic cate^ry. Racial categories for 
which National Assessment reports results 
are white, black and Hispanic. 

Receipt controL Procedures implemented by 
scoring staff to check in and screen mate- 
rials from the field. Information gained 
from receipt control procedures is relayed 
to assessment administration staff so that 
any errors may be corrected. 

Recycled exercise The set of exercises that is 
kept secure from one assesonent to the 
next that wH] he uped to m^sure changes 
(growth, stability or decline) in per- 
formance for the learning area. 

Region, One of four geographical regions used 
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in gathering and reporting data: Northeast, 
Southeast, Central and West, States 
inciuded in each r^on are shown in 
Appendix B. 

Released exercise. An exercise for which re- 
sults and exercise text have been reported 
to the public. 

Released exercise set, A set of released exer- 
cises, including documentation and scoring 
guides, made available to state and local 
education agenci^ and the r^arch , 
community. 

Reporting groups. Categories of variables for 
which National Ass^sment data are 
reported. Variable categories are deHned in 
Appendix B. 

Rescore. To insure that responses to open- 
ended items from both assessments of 
mathematics were scored in exactly the 
same manner, all responses to open-end^ 
items from the 1972—73 assessment were 
scored again at the same time as the 
1977—7^ responses to the^e items were 
scored. 

Respondent A person who responds to the 
exercises in a:i assessment booklet. 

Response options. Different alternatives to a 
multiple -choice question that can be 
selected fay the respondent. 

Review conference. A conference held to 
review the objectives of a learning area to 
assure their acceptance as measures of the 
objectives by scholars, educators and hy 
persons or to review exercises for radal^ 
ethnic, social or regional tu^. 

Sample, National Assessment does not assess 
an entire age population but ratl^r selects a 
representative sai. pie from the a^ group 
to answer assessment items. (See Chapter 4 
for a description of Naticmal Assessment 
sampling procedures.) 

Sconr^ guide. A guide for hand scoring an 



open-ended exercise that specifies descrip- 
tive or diagn<^c categories by giving 
deHnitions and example response. Cate- 
gories are usually defined as ''acceptable*' 
or "unacceptable." 

Size of community, Results are reported for 
four size^of -community cat^ories: big 
cities, fringes around big cities, medium 
cities and small places. These categories are 
defin^ in Appendix B. 

SMS A. Standard Metropolitan Statistical 
Area SMSAs are economic and social units 
defined by the U.S. Bureau of the Census. 

Stem. The portion of an exe^rcise that states 
the problem or asks the question. 

Stimuius. See stem. 

Subject ai^a. See learning area. 

Subpopulation or subgroup. Grouf^ within 
the national population for which data are 
reported. 

Supplementary principal's qu&tionnaij^. A 
data collection form given to principals. On 
this form, principals are asked to respond 
to questions concerning course offerings, 
materials and staffing specific to the learn- 
ing area being assessed. See also principaFs 
questionnaire. A sample of this ques- 
tionnaire is found in Appendix C. 

Tapescript. A script prepared for the an^ 
nouncer to use •in producing the paced 
tape. It indicates exactly what is to be read 
or not read and indicates the amount of 
response time allowed for ^ch exercise. 
See paced audio tape. 

Type of community. Results are reported for 
three type-of -community categories: dis- 
advantaged urban, advantaged inrban and 
extreme mral. Defiiiitiom of th^e cat^o- 
ries are found in Appendix B. 

Variable. A cla5sifi<mtion of respondents. 
Standard reporting variables are: region, 
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sex, race, level of parental education, size 
of community, type of community and 
grade in school. 

Weight. A multiplicative factor equal to the 
'reciprocal of the probability of a re- 
spondent being selected for assessment 



with ac^usteient for nonresponse — go es- 
timate of the number of persons in the 
population represented by a respondent in 
the sample. TheoreticaUy, the sum of 
wei^ts for all r^}ondepts at an ai^ level is 
equal to the number of persons in the 
country at that age level'. 
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